2024-08-31T21:59:40.459 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-08-31T21:59:40.478 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-31T21:59:40.565 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-08-31_21:16:02-rbd-squid-distro-default-smithi/7883608 branch: squid description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} email: ceph-qa@ceph.com first_in_suite: false job_id: '7883608' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: teuthology-2024-08-31_21:16:02-rbd-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.4: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.5: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.6: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 3091dfc0187f68df2afd4e392dc4c381cfdc1cad ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 3091dfc0187f68df2afd4e392dc4c381cfdc1cad thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 3091dfc0187f68df2afd4e392dc4c381cfdc1cad owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 552 sha1: 3091dfc0187f68df2afd4e392dc4c381cfdc1cad sleep_before_teardown: 0 subset: 40/128 suite: rbd suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_3091dfc0187f68df2afd4e392dc4c381cfdc1cad/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 3091dfc0187f68df2afd4e392dc4c381cfdc1cad targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPXUeOWcSMUCUypX4YCMcgYYBi9GKauqGGUmH0fi5vzIeU6SJbiht+tepppB/MtJQna3sVZ2fKQUW0qywuwh218= smithi032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC9ImDX2yUmt3UPP3yhZ5c5Z5ePr116mvTICgiHMVQJB4QGQf95OWkNXcWQYcyswUbvK1EMz1+9d/gLndRc9F3M= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' RBD_MIRROR_MIN_COMPAT_CLIENT: octopus RBD_MIRROR_USE_EXISTING_CLUSTER: '1' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-08-31_21:16:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-31T21:59:40.566 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_3091dfc0187f68df2afd4e392dc4c381cfdc1cad/qa; will attempt to use it 2024-08-31T21:59:40.566 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_3091dfc0187f68df2afd4e392dc4c381cfdc1cad/qa/tasks 2024-08-31T21:59:40.566 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-31T21:59:40.568 INFO:teuthology.task.internal:Checking packages... 2024-08-31T21:59:40.588 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '3091dfc0187f68df2afd4e392dc4c381cfdc1cad' 2024-08-31T21:59:40.588 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-31T21:59:40.589 INFO:teuthology.packaging:ref: None 2024-08-31T21:59:40.589 INFO:teuthology.packaging:tag: None 2024-08-31T21:59:40.589 INFO:teuthology.packaging:branch: squid 2024-08-31T21:59:40.589 INFO:teuthology.packaging:sha1: 3091dfc0187f68df2afd4e392dc4c381cfdc1cad 2024-08-31T21:59:40.589 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-08-31T21:59:40.784 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-244-g3091dfc0-1jammy 2024-08-31T21:59:40.786 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-31T21:59:40.792 INFO:teuthology.task.internal:no buildpackages task found 2024-08-31T21:59:40.792 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-31T21:59:40.805 INFO:teuthology.task.internal:Saving configuration 2024-08-31T21:59:40.821 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-31T21:59:40.831 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-31T21:59:40.849 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-31_21:16:02-rbd-squid-distro-default-smithi/7883608', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-31 21:54:13.219603', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINyDjZNaF1NTHLzzPC3dxmLchuBcFihTIobJvzb1uRqq'} 2024-08-31T21:59:40.866 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-31_21:16:02-rbd-squid-distro-default-smithi/7883608', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-31 21:54:13.221140', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOQ49ZCh8hYGH0tk+yavvjB1x9zXbbTJ0MU3sDDeT/hr'} 2024-08-31T21:59:40.866 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-31T21:59:40.872 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-08-31T21:59:40.873 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-08-31T21:59:40.873 INFO:teuthology.run_tasks:Running task console_log... 2024-08-31T21:59:40.922 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc362b07760>, signals=[15]) 2024-08-31T21:59:40.922 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-31T21:59:40.928 INFO:teuthology.task.internal:Opening connections... 2024-08-31T21:59:40.928 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2024-08-31T21:59:40.930 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-31T21:59:41.011 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2024-08-31T21:59:41.012 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-31T21:59:41.089 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-31T21:59:41.096 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2024-08-31T21:59:41.101 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2024-08-31T21:59:41.102 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2024-08-31T21:59:41.146 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-08-31T21:59:41.146 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="Ubuntu" 2024-08-31T21:59:41.146 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="22.04" 2024-08-31T21:59:41.146 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-08-31T21:59:41.146 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_CODENAME=jammy 2024-08-31T21:59:41.146 INFO:teuthology.orchestra.run.smithi007.stdout:ID=ubuntu 2024-08-31T21:59:41.147 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE=debian 2024-08-31T21:59:41.147 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://www.ubuntu.com/" 2024-08-31T21:59:41.147 INFO:teuthology.orchestra.run.smithi007.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-08-31T21:59:41.147 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-08-31T21:59:41.147 INFO:teuthology.orchestra.run.smithi007.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-08-31T21:59:41.147 INFO:teuthology.orchestra.run.smithi007.stdout:UBUNTU_CODENAME=jammy 2024-08-31T21:59:41.147 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2024-08-31T21:59:41.163 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2024-08-31T21:59:41.168 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2024-08-31T21:59:41.168 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2024-08-31T21:59:41.220 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-08-31T21:59:41.220 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="Ubuntu" 2024-08-31T21:59:41.220 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="22.04" 2024-08-31T21:59:41.220 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-08-31T21:59:41.220 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_CODENAME=jammy 2024-08-31T21:59:41.220 INFO:teuthology.orchestra.run.smithi032.stdout:ID=ubuntu 2024-08-31T21:59:41.221 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE=debian 2024-08-31T21:59:41.221 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://www.ubuntu.com/" 2024-08-31T21:59:41.221 INFO:teuthology.orchestra.run.smithi032.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-08-31T21:59:41.221 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-08-31T21:59:41.221 INFO:teuthology.orchestra.run.smithi032.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-08-31T21:59:41.221 INFO:teuthology.orchestra.run.smithi032.stdout:UBUNTU_CODENAME=jammy 2024-08-31T21:59:41.221 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2024-08-31T21:59:41.238 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-31T21:59:41.246 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-31T21:59:41.253 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-31T21:59:41.253 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2024-08-31T21:59:41.256 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2024-08-31T21:59:41.267 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-31T21:59:41.275 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-31T21:59:41.275 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2024-08-31T21:59:41.304 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2024-08-31T21:59:41.330 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-31T21:59:41.386 INFO:teuthology.run_tasks:Running task kernel... 2024-08-31T21:59:41.400 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-08-31T21:59:41.400 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-08-31T21:59:41.401 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-31T21:59:41.401 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-31T21:59:41.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T21:59:41.406 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-08-31T21:59:41.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T21:59:41.407 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-08-31T21:59:41.453 INFO:teuthology.orchestra.run.smithi032.stdout:5.15.0-56-generic 2024-08-31T21:59:41.453 INFO:teuthology.task.kernel:Running kernel on smithi032: 5.15.0-56-generic 2024-08-31T21:59:41.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo apt-get clean 2024-08-31T21:59:41.455 INFO:teuthology.orchestra.run.smithi007.stdout:5.15.0-56-generic 2024-08-31T21:59:41.455 INFO:teuthology.task.kernel:Running kernel on smithi007: 5.15.0-56-generic 2024-08-31T21:59:41.455 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get clean 2024-08-31T21:59:41.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get update 2024-08-31T21:59:41.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo apt-get update 2024-08-31T21:59:41.724 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-08-31T21:59:41.833 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-08-31T21:59:41.895 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-08-31T21:59:41.897 INFO:teuthology.orchestra.run.smithi032.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-08-31T21:59:41.915 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-08-31T21:59:41.920 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,769 kB] 2024-08-31T21:59:41.938 INFO:teuthology.orchestra.run.smithi032.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-08-31T21:59:42.023 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [531 kB] 2024-08-31T21:59:42.028 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [290 kB] 2024-08-31T21:59:42.031 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.2 kB] 2024-08-31T21:59:42.032 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,308 kB] 2024-08-31T21:59:42.053 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.3 kB] 2024-08-31T21:59:42.053 INFO:teuthology.orchestra.run.smithi007.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [398 kB] 2024-08-31T21:59:42.058 INFO:teuthology.orchestra.run.smithi007.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [572 B] 2024-08-31T21:59:42.058 INFO:teuthology.orchestra.run.smithi007.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [622 kB] 2024-08-31T21:59:42.064 INFO:teuthology.orchestra.run.smithi007.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [890 kB] 2024-08-31T21:59:42.071 INFO:teuthology.orchestra.run.smithi007.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [175 kB] 2024-08-31T21:59:42.073 INFO:teuthology.orchestra.run.smithi007.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.0 kB] 2024-08-31T21:59:42.073 INFO:teuthology.orchestra.run.smithi007.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-08-31T21:59:42.073 INFO:teuthology.orchestra.run.smithi007.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-08-31T21:59:42.084 INFO:teuthology.orchestra.run.smithi007.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-08-31T21:59:42.084 INFO:teuthology.orchestra.run.smithi007.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-08-31T21:59:42.271 INFO:teuthology.orchestra.run.smithi007.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-08-31T21:59:42.295 INFO:teuthology.orchestra.run.smithi032.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-08-31T21:59:42.317 INFO:teuthology.orchestra.run.smithi032.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,769 kB] 2024-08-31T21:59:42.409 INFO:teuthology.orchestra.run.smithi007.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-08-31T21:59:42.433 INFO:teuthology.orchestra.run.smithi032.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-08-31T21:59:42.603 INFO:teuthology.orchestra.run.smithi007.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,987 kB] 2024-08-31T21:59:42.638 INFO:teuthology.orchestra.run.smithi032.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,987 kB] 2024-08-31T21:59:42.754 INFO:teuthology.orchestra.run.smithi032.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [531 kB] 2024-08-31T21:59:42.762 INFO:teuthology.orchestra.run.smithi007.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-08-31T21:59:42.770 INFO:teuthology.orchestra.run.smithi032.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [290 kB] 2024-08-31T21:59:42.776 INFO:teuthology.orchestra.run.smithi007.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-08-31T21:59:42.777 INFO:teuthology.orchestra.run.smithi007.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,384 kB] 2024-08-31T21:59:42.780 INFO:teuthology.orchestra.run.smithi032.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.2 kB] 2024-08-31T21:59:42.780 INFO:teuthology.orchestra.run.smithi032.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,308 kB] 2024-08-31T21:59:42.785 INFO:teuthology.orchestra.run.smithi032.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-08-31T21:59:42.799 INFO:teuthology.orchestra.run.smithi032.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-08-31T21:59:42.801 INFO:teuthology.orchestra.run.smithi032.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.9 kB] 2024-08-31T21:59:42.802 INFO:teuthology.orchestra.run.smithi032.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,384 kB] 2024-08-31T21:59:42.844 INFO:teuthology.orchestra.run.smithi007.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.9 kB] 2024-08-31T21:59:42.845 INFO:teuthology.orchestra.run.smithi007.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-08-31T21:59:42.862 INFO:teuthology.orchestra.run.smithi007.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [604 B] 2024-08-31T21:59:42.862 INFO:teuthology.orchestra.run.smithi007.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,112 kB] 2024-08-31T21:59:42.870 INFO:teuthology.orchestra.run.smithi032.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-08-31T21:59:42.874 INFO:teuthology.orchestra.run.smithi032.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.3 kB] 2024-08-31T21:59:42.875 INFO:teuthology.orchestra.run.smithi032.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [398 kB] 2024-08-31T21:59:42.881 INFO:teuthology.orchestra.run.smithi032.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [604 B] 2024-08-31T21:59:42.897 INFO:teuthology.orchestra.run.smithi007.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [726 kB] 2024-08-31T21:59:42.905 INFO:teuthology.orchestra.run.smithi032.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,112 kB] 2024-08-31T21:59:42.905 INFO:teuthology.orchestra.run.smithi032.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [572 B] 2024-08-31T21:59:42.906 INFO:teuthology.orchestra.run.smithi032.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [622 kB] 2024-08-31T21:59:42.913 INFO:teuthology.orchestra.run.smithi007.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [260 kB] 2024-08-31T21:59:42.917 INFO:teuthology.orchestra.run.smithi032.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [890 kB] 2024-08-31T21:59:42.919 INFO:teuthology.orchestra.run.smithi007.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.0 kB] 2024-08-31T21:59:42.926 INFO:teuthology.orchestra.run.smithi032.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [726 kB] 2024-08-31T21:59:42.928 INFO:teuthology.orchestra.run.smithi007.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-08-31T21:59:42.929 INFO:teuthology.orchestra.run.smithi007.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-08-31T21:59:42.930 INFO:teuthology.orchestra.run.smithi007.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-08-31T21:59:42.931 INFO:teuthology.orchestra.run.smithi007.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-08-31T21:59:42.933 INFO:teuthology.orchestra.run.smithi032.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [175 kB] 2024-08-31T21:59:42.936 INFO:teuthology.orchestra.run.smithi032.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.0 kB] 2024-08-31T21:59:42.936 INFO:teuthology.orchestra.run.smithi032.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-08-31T21:59:42.937 INFO:teuthology.orchestra.run.smithi032.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-08-31T21:59:42.939 INFO:teuthology.orchestra.run.smithi032.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [260 kB] 2024-08-31T21:59:42.944 INFO:teuthology.orchestra.run.smithi032.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.0 kB] 2024-08-31T21:59:42.950 INFO:teuthology.orchestra.run.smithi032.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-08-31T21:59:42.951 INFO:teuthology.orchestra.run.smithi032.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-08-31T21:59:42.951 INFO:teuthology.orchestra.run.smithi032.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-08-31T21:59:43.003 INFO:teuthology.orchestra.run.smithi007.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-08-31T21:59:43.004 INFO:teuthology.orchestra.run.smithi007.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-08-31T21:59:43.005 INFO:teuthology.orchestra.run.smithi007.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-08-31T21:59:43.006 INFO:teuthology.orchestra.run.smithi007.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-08-31T21:59:43.006 INFO:teuthology.orchestra.run.smithi007.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-08-31T21:59:43.009 INFO:teuthology.orchestra.run.smithi007.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-08-31T21:59:43.015 INFO:teuthology.orchestra.run.smithi032.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-08-31T21:59:43.016 INFO:teuthology.orchestra.run.smithi032.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-08-31T21:59:43.026 INFO:teuthology.orchestra.run.smithi032.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-08-31T21:59:43.027 INFO:teuthology.orchestra.run.smithi032.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-08-31T21:59:43.028 INFO:teuthology.orchestra.run.smithi032.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-08-31T21:59:43.030 INFO:teuthology.orchestra.run.smithi032.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-08-31T21:59:43.030 INFO:teuthology.orchestra.run.smithi032.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-08-31T21:59:43.031 INFO:teuthology.orchestra.run.smithi032.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-08-31T21:59:43.032 INFO:teuthology.orchestra.run.smithi032.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-08-31T21:59:43.091 INFO:teuthology.orchestra.run.smithi007.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-08-31T21:59:43.092 INFO:teuthology.orchestra.run.smithi007.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-08-31T21:59:43.113 INFO:teuthology.orchestra.run.smithi032.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-08-31T21:59:43.114 INFO:teuthology.orchestra.run.smithi032.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-08-31T21:59:48.751 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 15.7 MB in 2s (7,283 kB/s) 2024-08-31T21:59:49.428 INFO:teuthology.orchestra.run.smithi032.stdout:Fetched 15.7 MB in 3s (5,644 kB/s) 2024-08-31T21:59:50.150 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2024-08-31T21:59:50.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-08-31T21:59:50.237 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2024-08-31T21:59:50.451 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2024-08-31T21:59:50.452 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2024-08-31T21:59:50.646 INFO:teuthology.orchestra.run.smithi007.stdout:The following packages were automatically installed and are no longer required: 2024-08-31T21:59:50.646 INFO:teuthology.orchestra.run.smithi007.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-08-31T21:59:50.646 INFO:teuthology.orchestra.run.smithi007.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-08-31T21:59:50.647 INFO:teuthology.orchestra.run.smithi007.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-08-31T21:59:50.647 INFO:teuthology.orchestra.run.smithi007.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-08-31T21:59:50.647 INFO:teuthology.orchestra.run.smithi007.stdout: python2.7-minimal 2024-08-31T21:59:50.647 INFO:teuthology.orchestra.run.smithi007.stdout:Use 'sudo apt autoremove' to remove them. 2024-08-31T21:59:50.648 INFO:teuthology.orchestra.run.smithi007.stdout:The following additional packages will be installed: 2024-08-31T21:59:50.648 INFO:teuthology.orchestra.run.smithi007.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-08-31T21:59:50.648 INFO:teuthology.orchestra.run.smithi007.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-08-31T21:59:50.649 INFO:teuthology.orchestra.run.smithi007.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-08-31T21:59:50.650 INFO:teuthology.orchestra.run.smithi007.stdout:Suggested packages: 2024-08-31T21:59:50.650 INFO:teuthology.orchestra.run.smithi007.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-08-31T21:59:50.650 INFO:teuthology.orchestra.run.smithi007.stdout:Recommended packages: 2024-08-31T21:59:50.650 INFO:teuthology.orchestra.run.smithi007.stdout: thermald 2024-08-31T21:59:50.708 INFO:teuthology.orchestra.run.smithi007.stdout:The following NEW packages will be installed: 2024-08-31T21:59:50.709 INFO:teuthology.orchestra.run.smithi007.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-08-31T21:59:50.709 INFO:teuthology.orchestra.run.smithi007.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-08-31T21:59:50.709 INFO:teuthology.orchestra.run.smithi007.stdout: linux-modules-extra-5.15.0-119-generic 2024-08-31T21:59:50.710 INFO:teuthology.orchestra.run.smithi007.stdout:The following packages will be upgraded: 2024-08-31T21:59:50.711 INFO:teuthology.orchestra.run.smithi007.stdout: linux-generic linux-headers-generic linux-image-generic 2024-08-31T21:59:50.763 INFO:teuthology.orchestra.run.smithi007.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-08-31T21:59:50.764 INFO:teuthology.orchestra.run.smithi007.stdout:Need to get 113 MB of archives. 2024-08-31T21:59:50.764 INFO:teuthology.orchestra.run.smithi007.stdout:After this operation, 583 MB of additional disk space will be used. 2024-08-31T21:59:50.764 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-08-31T21:59:51.025 INFO:teuthology.orchestra.run.smithi032.stdout:Reading package lists... 2024-08-31T21:59:51.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-08-31T21:59:51.101 INFO:teuthology.orchestra.run.smithi032.stdout:Reading package lists... 2024-08-31T21:59:51.196 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-08-31T21:59:51.295 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-08-31T21:59:51.311 INFO:teuthology.orchestra.run.smithi032.stdout:Building dependency tree... 2024-08-31T21:59:51.311 INFO:teuthology.orchestra.run.smithi032.stdout:Reading state information... 2024-08-31T21:59:51.483 INFO:teuthology.orchestra.run.smithi032.stdout:The following packages were automatically installed and are no longer required: 2024-08-31T21:59:51.483 INFO:teuthology.orchestra.run.smithi032.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-08-31T21:59:51.484 INFO:teuthology.orchestra.run.smithi032.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-08-31T21:59:51.484 INFO:teuthology.orchestra.run.smithi032.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-08-31T21:59:51.484 INFO:teuthology.orchestra.run.smithi032.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-08-31T21:59:51.484 INFO:teuthology.orchestra.run.smithi032.stdout: python2.7-minimal 2024-08-31T21:59:51.484 INFO:teuthology.orchestra.run.smithi032.stdout:Use 'sudo apt autoremove' to remove them. 2024-08-31T21:59:51.485 INFO:teuthology.orchestra.run.smithi032.stdout:The following additional packages will be installed: 2024-08-31T21:59:51.485 INFO:teuthology.orchestra.run.smithi032.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-08-31T21:59:51.485 INFO:teuthology.orchestra.run.smithi032.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-08-31T21:59:51.486 INFO:teuthology.orchestra.run.smithi032.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-08-31T21:59:51.487 INFO:teuthology.orchestra.run.smithi032.stdout:Suggested packages: 2024-08-31T21:59:51.487 INFO:teuthology.orchestra.run.smithi032.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-08-31T21:59:51.487 INFO:teuthology.orchestra.run.smithi032.stdout:Recommended packages: 2024-08-31T21:59:51.487 INFO:teuthology.orchestra.run.smithi032.stdout: thermald 2024-08-31T21:59:51.542 INFO:teuthology.orchestra.run.smithi032.stdout:The following NEW packages will be installed: 2024-08-31T21:59:51.542 INFO:teuthology.orchestra.run.smithi032.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-08-31T21:59:51.542 INFO:teuthology.orchestra.run.smithi032.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-08-31T21:59:51.542 INFO:teuthology.orchestra.run.smithi032.stdout: linux-modules-extra-5.15.0-119-generic 2024-08-31T21:59:51.543 INFO:teuthology.orchestra.run.smithi032.stdout:The following packages will be upgraded: 2024-08-31T21:59:51.544 INFO:teuthology.orchestra.run.smithi032.stdout: linux-generic linux-headers-generic linux-image-generic 2024-08-31T21:59:51.724 INFO:teuthology.orchestra.run.smithi032.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-08-31T21:59:51.724 INFO:teuthology.orchestra.run.smithi032.stdout:Need to get 113 MB of archives. 2024-08-31T21:59:51.724 INFO:teuthology.orchestra.run.smithi032.stdout:After this operation, 583 MB of additional disk space will be used. 2024-08-31T21:59:51.724 INFO:teuthology.orchestra.run.smithi032.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-08-31T21:59:52.120 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-08-31T21:59:52.120 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-08-31T21:59:52.120 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-08-31T21:59:52.248 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-08-31T21:59:52.271 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-08-31T21:59:52.668 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 113 MB in 2s (73.0 MB/s) 2024-08-31T21:59:52.827 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-08-31T21:59:53.114 INFO:teuthology.orchestra.run.smithi032.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-08-31T21:59:53.509 INFO:teuthology.orchestra.run.smithi032.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-08-31T21:59:54.813 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-08-31T21:59:54.818 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-08-31T21:59:54.975 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-31T21:59:55.671 INFO:teuthology.orchestra.run.smithi032.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-08-31T21:59:55.671 INFO:teuthology.orchestra.run.smithi032.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-08-31T21:59:55.671 INFO:teuthology.orchestra.run.smithi032.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-08-31T21:59:56.018 INFO:teuthology.orchestra.run.smithi032.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-08-31T21:59:56.097 INFO:teuthology.orchestra.run.smithi032.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-08-31T21:59:56.516 INFO:teuthology.orchestra.run.smithi032.stdout:Fetched 113 MB in 5s (24.9 MB/s) 2024-08-31T21:59:56.711 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-08-31T21:59:57.027 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-08-31T21:59:57.048 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-08-31T21:59:57.143 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-31T21:59:57.555 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-08-31T21:59:57.565 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-08-31T21:59:57.597 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-31T21:59:58.704 INFO:teuthology.orchestra.run.smithi032.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-08-31T21:59:58.708 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-08-31T21:59:58.951 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-31T22:00:01.137 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-08-31T22:00:01.159 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-08-31T22:00:01.272 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-31T22:00:01.797 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-08-31T22:00:01.818 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-08-31T22:00:02.137 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-31T22:00:03.591 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-08-31T22:00:03.718 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-08-31T22:00:04.050 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-08-31T22:00:04.163 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-08-31T22:00:04.433 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-08-31T22:00:04.444 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-08-31T22:00:04.483 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-08-31T22:00:08.965 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-08-31T22:00:09.114 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-08-31T22:00:09.465 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-08-31T22:00:09.593 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-08-31T22:00:09.733 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-08-31T22:00:09.757 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-08-31T22:00:09.799 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-31T22:00:09.913 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-08-31T22:00:09.936 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-08-31T22:00:10.005 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-08-31T22:00:12.894 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-08-31T22:00:13.002 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-08-31T22:00:13.320 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-08-31T22:00:13.448 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-31T22:00:13.575 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-08-31T22:00:13.703 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-31T22:00:15.021 INFO:teuthology.orchestra.run.smithi007.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-08-31T22:00:15.021 INFO:teuthology.orchestra.run.smithi007.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-08-31T22:00:15.021 INFO:teuthology.orchestra.run.smithi007.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-08-31T22:00:15.021 INFO:teuthology.orchestra.run.smithi007.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-08-31T22:00:15.188 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-31T22:00:15.685 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-08-31T22:00:15.723 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-08-31T22:00:15.767 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-31T22:00:16.647 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-31T22:00:18.407 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-08-31T22:00:18.533 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-08-31T22:00:18.659 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-31T22:00:18.714 INFO:teuthology.orchestra.run.smithi007.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-08-31T22:00:18.714 INFO:teuthology.orchestra.run.smithi007.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-08-31T22:00:20.296 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-08-31T22:00:20.453 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-08-31T22:00:20.821 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-08-31T22:00:20.941 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-31T22:00:21.093 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-08-31T22:00:21.222 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-31T22:00:22.570 INFO:teuthology.orchestra.run.smithi032.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-08-31T22:00:22.570 INFO:teuthology.orchestra.run.smithi032.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-08-31T22:00:22.570 INFO:teuthology.orchestra.run.smithi032.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-08-31T22:00:22.570 INFO:teuthology.orchestra.run.smithi032.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-08-31T22:00:22.740 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-31T22:00:24.716 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-31T22:00:26.134 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-08-31T22:00:26.302 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-08-31T22:00:26.428 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-31T22:00:26.491 INFO:teuthology.orchestra.run.smithi032.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-08-31T22:00:26.491 INFO:teuthology.orchestra.run.smithi032.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-08-31T22:00:33.712 INFO:teuthology.orchestra.run.smithi007.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-08-31T22:00:33.712 INFO:teuthology.orchestra.run.smithi007.stdout:Sourcing file `/etc/default/grub' 2024-08-31T22:00:33.728 INFO:teuthology.orchestra.run.smithi007.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-08-31T22:00:33.750 INFO:teuthology.orchestra.run.smithi007.stdout:Generating grub configuration file ... 2024-08-31T22:00:33.987 INFO:teuthology.orchestra.run.smithi007.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-08-31T22:00:34.005 INFO:teuthology.orchestra.run.smithi007.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-08-31T22:00:34.269 INFO:teuthology.orchestra.run.smithi007.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-08-31T22:00:34.276 INFO:teuthology.orchestra.run.smithi007.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-08-31T22:00:34.352 INFO:teuthology.orchestra.run.smithi007.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-08-31T22:00:34.358 INFO:teuthology.orchestra.run.smithi007.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-08-31T22:00:34.521 INFO:teuthology.orchestra.run.smithi007.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-08-31T22:00:34.521 INFO:teuthology.orchestra.run.smithi007.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-08-31T22:00:34.521 INFO:teuthology.orchestra.run.smithi007.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-08-31T22:00:34.574 INFO:teuthology.orchestra.run.smithi007.stdout:done 2024-08-31T22:00:35.032 DEBUG:teuthology.orchestra.run.smithi007:> dpkg -s linux-image-generic 2024-08-31T22:00:35.050 INFO:teuthology.orchestra.run.smithi007.stdout:Package: linux-image-generic 2024-08-31T22:00:35.050 INFO:teuthology.orchestra.run.smithi007.stdout:Status: install ok installed 2024-08-31T22:00:35.050 INFO:teuthology.orchestra.run.smithi007.stdout:Priority: optional 2024-08-31T22:00:35.050 INFO:teuthology.orchestra.run.smithi007.stdout:Section: kernel 2024-08-31T22:00:35.050 INFO:teuthology.orchestra.run.smithi007.stdout:Installed-Size: 21 2024-08-31T22:00:35.050 INFO:teuthology.orchestra.run.smithi007.stdout:Maintainer: Ubuntu Kernel Team 2024-08-31T22:00:35.051 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: amd64 2024-08-31T22:00:35.051 INFO:teuthology.orchestra.run.smithi007.stdout:Source: linux-meta 2024-08-31T22:00:35.051 INFO:teuthology.orchestra.run.smithi007.stdout:Version: 5.15.0.119.119 2024-08-31T22:00:35.051 INFO:teuthology.orchestra.run.smithi007.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-08-31T22:00:35.051 INFO:teuthology.orchestra.run.smithi007.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-08-31T22:00:35.051 INFO:teuthology.orchestra.run.smithi007.stdout:Recommends: thermald 2024-08-31T22:00:35.051 INFO:teuthology.orchestra.run.smithi007.stdout:Description: Generic Linux kernel image 2024-08-31T22:00:35.051 INFO:teuthology.orchestra.run.smithi007.stdout: This package will always depend on the latest generic kernel image 2024-08-31T22:00:35.051 INFO:teuthology.orchestra.run.smithi007.stdout: available. 2024-08-31T22:00:35.051 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-08-31T22:00:35.052 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-31T22:00:35.052 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-31T22:00:35.052 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-08-31T22:00:35.052 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2024-08-31T22:00:35.052 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get clean 2024-08-31T22:00:35.195 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get update 2024-08-31T22:00:35.454 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-08-31T22:00:35.454 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-08-31T22:00:35.518 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-08-31T22:00:35.599 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-08-31T22:00:36.901 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2024-08-31T22:00:36.920 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-08-31T22:00:36.975 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2024-08-31T22:00:37.165 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2024-08-31T22:00:37.165 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2024-08-31T22:00:37.338 INFO:teuthology.orchestra.run.smithi007.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-08-31T22:00:37.338 INFO:teuthology.orchestra.run.smithi007.stdout:The following packages were automatically installed and are no longer required: 2024-08-31T22:00:37.338 INFO:teuthology.orchestra.run.smithi007.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-08-31T22:00:37.338 INFO:teuthology.orchestra.run.smithi007.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-08-31T22:00:37.338 INFO:teuthology.orchestra.run.smithi007.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-08-31T22:00:37.339 INFO:teuthology.orchestra.run.smithi007.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-08-31T22:00:37.339 INFO:teuthology.orchestra.run.smithi007.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-08-31T22:00:37.339 INFO:teuthology.orchestra.run.smithi007.stdout:Use 'sudo apt autoremove' to remove them. 2024-08-31T22:00:37.380 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-08-31T22:00:37.383 DEBUG:teuthology.orchestra.run.smithi007:> dpkg -s linux-image-generic 2024-08-31T22:00:37.441 INFO:teuthology.orchestra.run.smithi007.stdout:Package: linux-image-generic 2024-08-31T22:00:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:Status: install ok installed 2024-08-31T22:00:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:Priority: optional 2024-08-31T22:00:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:Section: kernel 2024-08-31T22:00:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:Installed-Size: 21 2024-08-31T22:00:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:Maintainer: Ubuntu Kernel Team 2024-08-31T22:00:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: amd64 2024-08-31T22:00:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:Source: linux-meta 2024-08-31T22:00:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:Version: 5.15.0.119.119 2024-08-31T22:00:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-08-31T22:00:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-08-31T22:00:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:Recommends: thermald 2024-08-31T22:00:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:Description: Generic Linux kernel image 2024-08-31T22:00:37.442 INFO:teuthology.orchestra.run.smithi007.stdout: This package will always depend on the latest generic kernel image 2024-08-31T22:00:37.442 INFO:teuthology.orchestra.run.smithi007.stdout: available. 2024-08-31T22:00:37.443 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-08-31T22:00:37.489 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.dUVYV6J9Za 2024-08-31T22:00:37.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub/grub.cfg /tmp/tmp.dUVYV6J9Za 2024-08-31T22:00:37.541 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.dUVYV6J9Za 2024-08-31T22:00:37.666 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.dUVYV6J9Za is 10KB 2024-08-31T22:00:37.678 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.dUVYV6J9Za 2024-08-31T22:00:37.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-08-31T22:00:37.737 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:00:37.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-08-31T22:00:37.738 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-08-31T22:00:37.828 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-08-31T22:00:37.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo update-grub 2024-08-31T22:00:39.001 INFO:teuthology.orchestra.run.smithi007.stderr:Sourcing file `/etc/default/grub' 2024-08-31T22:00:39.002 INFO:teuthology.orchestra.run.smithi007.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-08-31T22:00:39.010 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2024-08-31T22:00:39.200 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-08-31T22:00:39.206 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-08-31T22:00:39.414 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-08-31T22:00:39.420 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-08-31T22:00:39.462 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-08-31T22:00:39.467 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-08-31T22:00:39.590 INFO:teuthology.orchestra.run.smithi007.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-08-31T22:00:39.590 INFO:teuthology.orchestra.run.smithi007.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-08-31T22:00:39.590 INFO:teuthology.orchestra.run.smithi007.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-08-31T22:00:39.617 INFO:teuthology.orchestra.run.smithi007.stderr:done 2024-08-31T22:00:39.634 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2024-08-31T22:00:41.242 INFO:teuthology.orchestra.run.smithi032.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-08-31T22:00:41.243 INFO:teuthology.orchestra.run.smithi032.stdout:Sourcing file `/etc/default/grub' 2024-08-31T22:00:41.270 INFO:teuthology.orchestra.run.smithi032.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-08-31T22:00:41.286 INFO:teuthology.orchestra.run.smithi032.stdout:Generating grub configuration file ... 2024-08-31T22:00:41.602 INFO:teuthology.orchestra.run.smithi032.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-08-31T22:00:41.630 INFO:teuthology.orchestra.run.smithi032.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-08-31T22:00:41.946 INFO:teuthology.orchestra.run.smithi032.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-08-31T22:00:41.951 INFO:teuthology.orchestra.run.smithi032.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-08-31T22:00:42.024 INFO:teuthology.orchestra.run.smithi032.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-08-31T22:00:42.030 INFO:teuthology.orchestra.run.smithi032.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-08-31T22:00:42.189 INFO:teuthology.orchestra.run.smithi032.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-08-31T22:00:42.189 INFO:teuthology.orchestra.run.smithi032.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-08-31T22:00:42.189 INFO:teuthology.orchestra.run.smithi032.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-08-31T22:00:42.248 INFO:teuthology.orchestra.run.smithi032.stdout:done 2024-08-31T22:00:42.716 DEBUG:teuthology.orchestra.run.smithi032:> dpkg -s linux-image-generic 2024-08-31T22:00:42.738 INFO:teuthology.orchestra.run.smithi032.stdout:Package: linux-image-generic 2024-08-31T22:00:42.739 INFO:teuthology.orchestra.run.smithi032.stdout:Status: install ok installed 2024-08-31T22:00:42.739 INFO:teuthology.orchestra.run.smithi032.stdout:Priority: optional 2024-08-31T22:00:42.739 INFO:teuthology.orchestra.run.smithi032.stdout:Section: kernel 2024-08-31T22:00:42.739 INFO:teuthology.orchestra.run.smithi032.stdout:Installed-Size: 21 2024-08-31T22:00:42.739 INFO:teuthology.orchestra.run.smithi032.stdout:Maintainer: Ubuntu Kernel Team 2024-08-31T22:00:42.739 INFO:teuthology.orchestra.run.smithi032.stdout:Architecture: amd64 2024-08-31T22:00:42.739 INFO:teuthology.orchestra.run.smithi032.stdout:Source: linux-meta 2024-08-31T22:00:42.739 INFO:teuthology.orchestra.run.smithi032.stdout:Version: 5.15.0.119.119 2024-08-31T22:00:42.739 INFO:teuthology.orchestra.run.smithi032.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-08-31T22:00:42.739 INFO:teuthology.orchestra.run.smithi032.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-08-31T22:00:42.739 INFO:teuthology.orchestra.run.smithi032.stdout:Recommends: thermald 2024-08-31T22:00:42.739 INFO:teuthology.orchestra.run.smithi032.stdout:Description: Generic Linux kernel image 2024-08-31T22:00:42.740 INFO:teuthology.orchestra.run.smithi032.stdout: This package will always depend on the latest generic kernel image 2024-08-31T22:00:42.740 INFO:teuthology.orchestra.run.smithi032.stdout: available. 2024-08-31T22:00:42.740 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-08-31T22:00:42.740 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-31T22:00:42.740 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-31T22:00:42.740 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-08-31T22:00:42.740 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi032.front.sepia.ceph.com, path=None, version=distro) 2024-08-31T22:00:42.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo apt-get clean 2024-08-31T22:00:42.954 DEBUG:teuthology.orchestra.run.smithi032:> sudo apt-get update 2024-08-31T22:00:43.071 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-08-31T22:00:43.194 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-08-31T22:00:43.275 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-08-31T22:00:43.356 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-08-31T22:00:44.593 INFO:teuthology.orchestra.run.smithi032.stdout:Reading package lists... 2024-08-31T22:00:44.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-08-31T22:00:44.676 INFO:teuthology.orchestra.run.smithi032.stdout:Reading package lists... 2024-08-31T22:00:44.891 INFO:teuthology.orchestra.run.smithi032.stdout:Building dependency tree... 2024-08-31T22:00:44.891 INFO:teuthology.orchestra.run.smithi032.stdout:Reading state information... 2024-08-31T22:00:45.089 INFO:teuthology.orchestra.run.smithi032.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-08-31T22:00:45.089 INFO:teuthology.orchestra.run.smithi032.stdout:The following packages were automatically installed and are no longer required: 2024-08-31T22:00:45.089 INFO:teuthology.orchestra.run.smithi032.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-08-31T22:00:45.089 INFO:teuthology.orchestra.run.smithi032.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-08-31T22:00:45.089 INFO:teuthology.orchestra.run.smithi032.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-08-31T22:00:45.090 INFO:teuthology.orchestra.run.smithi032.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-08-31T22:00:45.090 INFO:teuthology.orchestra.run.smithi032.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-08-31T22:00:45.090 INFO:teuthology.orchestra.run.smithi032.stdout:Use 'sudo apt autoremove' to remove them. 2024-08-31T22:00:45.133 INFO:teuthology.orchestra.run.smithi032.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-08-31T22:00:45.135 DEBUG:teuthology.orchestra.run.smithi032:> dpkg -s linux-image-generic 2024-08-31T22:00:45.151 INFO:teuthology.orchestra.run.smithi032.stdout:Package: linux-image-generic 2024-08-31T22:00:45.151 INFO:teuthology.orchestra.run.smithi032.stdout:Status: install ok installed 2024-08-31T22:00:45.151 INFO:teuthology.orchestra.run.smithi032.stdout:Priority: optional 2024-08-31T22:00:45.151 INFO:teuthology.orchestra.run.smithi032.stdout:Section: kernel 2024-08-31T22:00:45.151 INFO:teuthology.orchestra.run.smithi032.stdout:Installed-Size: 21 2024-08-31T22:00:45.151 INFO:teuthology.orchestra.run.smithi032.stdout:Maintainer: Ubuntu Kernel Team 2024-08-31T22:00:45.151 INFO:teuthology.orchestra.run.smithi032.stdout:Architecture: amd64 2024-08-31T22:00:45.151 INFO:teuthology.orchestra.run.smithi032.stdout:Source: linux-meta 2024-08-31T22:00:45.151 INFO:teuthology.orchestra.run.smithi032.stdout:Version: 5.15.0.119.119 2024-08-31T22:00:45.152 INFO:teuthology.orchestra.run.smithi032.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-08-31T22:00:45.152 INFO:teuthology.orchestra.run.smithi032.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-08-31T22:00:45.152 INFO:teuthology.orchestra.run.smithi032.stdout:Recommends: thermald 2024-08-31T22:00:45.152 INFO:teuthology.orchestra.run.smithi032.stdout:Description: Generic Linux kernel image 2024-08-31T22:00:45.152 INFO:teuthology.orchestra.run.smithi032.stdout: This package will always depend on the latest generic kernel image 2024-08-31T22:00:45.152 INFO:teuthology.orchestra.run.smithi032.stdout: available. 2024-08-31T22:00:45.153 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-08-31T22:00:45.199 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.hydS8YBCqa 2024-08-31T22:00:45.200 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /boot/grub/grub.cfg /tmp/tmp.hydS8YBCqa 2024-08-31T22:00:45.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.hydS8YBCqa 2024-08-31T22:00:45.377 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.hydS8YBCqa is 10KB 2024-08-31T22:00:45.390 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.hydS8YBCqa 2024-08-31T22:00:45.395 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-08-31T22:00:45.451 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:00:45.451 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-08-31T22:00:45.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-08-31T22:00:45.534 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-08-31T22:00:45.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo update-grub 2024-08-31T22:00:46.854 INFO:teuthology.orchestra.run.smithi032.stderr:Sourcing file `/etc/default/grub' 2024-08-31T22:00:46.855 INFO:teuthology.orchestra.run.smithi032.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-08-31T22:00:46.857 INFO:teuthology.orchestra.run.smithi032.stderr:Generating grub configuration file ... 2024-08-31T22:00:47.087 INFO:teuthology.orchestra.run.smithi032.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-08-31T22:00:47.094 INFO:teuthology.orchestra.run.smithi032.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-08-31T22:00:47.376 INFO:teuthology.orchestra.run.smithi032.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-08-31T22:00:47.382 INFO:teuthology.orchestra.run.smithi032.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-08-31T22:00:47.424 INFO:teuthology.orchestra.run.smithi032.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-08-31T22:00:47.430 INFO:teuthology.orchestra.run.smithi032.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-08-31T22:00:47.540 INFO:teuthology.orchestra.run.smithi032.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-08-31T22:00:47.541 INFO:teuthology.orchestra.run.smithi032.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-08-31T22:00:47.541 INFO:teuthology.orchestra.run.smithi032.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-08-31T22:00:47.564 INFO:teuthology.orchestra.run.smithi032.stderr:done 2024-08-31T22:00:47.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo shutdown -r now 2024-08-31T22:01:09.657 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-08-31T22:01:09.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-08-31T22:01:09.658 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-31T22:01:17.576 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-08-31T22:01:17.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-08-31T22:01:17.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-31T22:01:36.061 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-08-31T22:01:45.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-08-31T22:01:45.068 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-31T22:02:03.453 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-08-31T22:02:09.665 DEBUG:teuthology.orchestra.remote:timed out 2024-08-31T22:02:15.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-08-31T22:02:15.458 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-31T22:02:17.501 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-08-31T22:02:18.668 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-08-31T22:02:18.668 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-31T22:02:32.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-08-31T22:02:32.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-31T22:02:32.783 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-08-31T22:02:33.495 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-08-31T22:02:33.495 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.15.0-119-generic"... 2024-08-31T22:02:33.495 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-08-31T22:02:33.540 INFO:teuthology.orchestra.run.smithi032.stdout:5.15.0-119-generic 2024-08-31T22:02:33.541 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-08-31T22:02:33.541 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-31T22:02:33.541 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-08-31T22:02:34.115 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-08-31T22:02:34.542 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-08-31T22:02:34.543 DEBUG:teuthology.orchestra.run.smithi032:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-31T22:02:34.617 INFO:teuthology.orchestra.run.smithi032.stdout:ttyS1 2024-08-31T22:02:34.646 DEBUG:teuthology.parallel:result is None 2024-08-31T22:02:34.663 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-08-31T22:02:34.663 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.15.0-119-generic"... 2024-08-31T22:02:34.663 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-08-31T22:02:34.709 INFO:teuthology.orchestra.run.smithi007.stdout:5.15.0-119-generic 2024-08-31T22:02:34.710 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-08-31T22:02:34.710 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-31T22:02:34.710 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-08-31T22:02:35.711 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-08-31T22:02:35.711 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-31T22:02:35.787 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2024-08-31T22:02:35.819 DEBUG:teuthology.parallel:result is None 2024-08-31T22:02:35.819 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-31T22:02:35.831 INFO:teuthology.task.internal:Creating test directory... 2024-08-31T22:02:35.831 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-31T22:02:35.833 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-31T22:02:35.839 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-31T22:02:35.846 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-31T22:02:35.855 INFO:teuthology.task.internal:Creating archive directory... 2024-08-31T22:02:35.856 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-31T22:02:35.882 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-31T22:02:35.902 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-31T22:02:35.909 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-31T22:02:35.909 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-31T22:02:35.938 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-31T22:02:35.959 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-31T22:02:35.962 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-31T22:02:35.968 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-31T22:02:35.970 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-31T22:02:35.972 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-31T22:02:35.981 INFO:teuthology.task.internal:Configuring sudo... 2024-08-31T22:02:35.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-31T22:02:36.017 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-31T22:02:36.029 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-31T22:02:36.038 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-31T22:02:36.038 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-31T22:02:36.066 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-31T22:02:36.076 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-31T22:02:36.122 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-31T22:02:36.170 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:02:36.170 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-31T22:02:36.233 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-31T22:02:36.239 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-31T22:02:36.285 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:02:36.285 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-31T22:02:36.349 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2024-08-31T22:02:36.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2024-08-31T22:02:36.425 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-31T22:02:36.432 INFO:teuthology.task.internal:Starting timer... 2024-08-31T22:02:36.432 INFO:teuthology.run_tasks:Running task pcp... 2024-08-31T22:02:36.445 INFO:teuthology.run_tasks:Running task selinux... 2024-08-31T22:02:36.453 DEBUG:teuthology.task.selinux:Excluding smithi007: OS 'ubuntu' does not support SELinux 2024-08-31T22:02:36.453 DEBUG:teuthology.task.selinux:Excluding smithi032: OS 'ubuntu' does not support SELinux 2024-08-31T22:02:36.453 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-31T22:02:36.453 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-08-31T22:02:36.453 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-31T22:02:36.453 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-31T22:02:36.461 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-08-31T22:02:36.464 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-08-31T22:02:36.588 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-31T22:02:36.599 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-31T22:02:36.600 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi032.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-31T22:10:17.837 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi032.front.sepia.ceph.com')] 2024-08-31T22:10:17.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-08-31T22:10:17.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-31T22:10:17.922 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-08-31T22:10:18.005 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-08-31T22:10:18.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-08-31T22:10:18.005 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-31T22:10:18.101 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-08-31T22:10:18.185 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-08-31T22:10:18.185 INFO:teuthology.run_tasks:Running task clock... 2024-08-31T22:10:18.195 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-31T22:10:18.195 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-31T22:10:18.196 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-31T22:10:18.198 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-31T22:10:18.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-31T22:10:18.227 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-08-31T22:10:18.227 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: Command line: ntpd -gq 2024-08-31T22:10:18.227 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: ---------------------------------------------------- 2024-08-31T22:10:18.227 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: ntp-4 is maintained by Network Time Foundation, 2024-08-31T22:10:18.227 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-08-31T22:10:18.227 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: corporation. Support and training for ntp-4 are 2024-08-31T22:10:18.227 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: available at https://www.nwtime.org/support 2024-08-31T22:10:18.227 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: ---------------------------------------------------- 2024-08-31T22:10:18.227 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: proto: precision = 0.061 usec (-24) 2024-08-31T22:10:18.228 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: basedate set to 2022-02-04 2024-08-31T22:10:18.228 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: gps base set to 2022-02-06 (week 2196) 2024-08-31T22:10:18.228 INFO:teuthology.orchestra.run.smithi007.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-08-31T22:10:18.229 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-08-31T22:10:18.229 INFO:teuthology.orchestra.run.smithi007.stderr:restrict ::: KOD does nothing without LIMITED. 2024-08-31T22:10:18.230 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: restrict ::: KOD does nothing without LIMITED. 2024-08-31T22:10:18.230 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: Listen and drop on 0 v6wildcard [::]:123 2024-08-31T22:10:18.230 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-08-31T22:10:18.230 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: Listen normally on 2 lo 127.0.0.1:123 2024-08-31T22:10:18.230 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: Listen normally on 3 ens1f0 172.21.15.7:123 2024-08-31T22:10:18.230 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: Listen normally on 4 lo [::1]:123 2024-08-31T22:10:18.230 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15ea%4]:123 2024-08-31T22:10:18.230 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:18 ntpd[18325]: Listening on routing socket on fd #22 for interface updates 2024-08-31T22:10:18.260 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-08-31T22:10:18.261 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: Command line: ntpd -gq 2024-08-31T22:10:18.261 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: ---------------------------------------------------- 2024-08-31T22:10:18.261 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: ntp-4 is maintained by Network Time Foundation, 2024-08-31T22:10:18.261 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-08-31T22:10:18.261 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: corporation. Support and training for ntp-4 are 2024-08-31T22:10:18.261 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: available at https://www.nwtime.org/support 2024-08-31T22:10:18.261 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: ---------------------------------------------------- 2024-08-31T22:10:18.262 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: proto: precision = 0.064 usec (-24) 2024-08-31T22:10:18.262 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: basedate set to 2022-02-04 2024-08-31T22:10:18.262 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: gps base set to 2022-02-06 (week 2196) 2024-08-31T22:10:18.262 INFO:teuthology.orchestra.run.smithi032.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-08-31T22:10:18.262 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-08-31T22:10:18.262 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: restrict ::: KOD does nothing without LIMITED. 2024-08-31T22:10:18.263 INFO:teuthology.orchestra.run.smithi032.stderr:restrict ::: KOD does nothing without LIMITED. 2024-08-31T22:10:18.264 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: Listen and drop on 0 v6wildcard [::]:123 2024-08-31T22:10:18.264 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-08-31T22:10:18.264 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: Listen normally on 2 lo 127.0.0.1:123 2024-08-31T22:10:18.264 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: Listen normally on 3 ens1f0 172.21.15.32:123 2024-08-31T22:10:18.264 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: Listen normally on 4 lo [::1]:123 2024-08-31T22:10:18.264 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fef8:2602%4]:123 2024-08-31T22:10:18.264 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:18 ntpd[18274]: Listening on routing socket on fd #22 for interface updates 2024-08-31T22:10:19.227 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:19 ntpd[18325]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:19.227 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:19 ntpd[18325]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:19.227 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:19 ntpd[18325]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:19.227 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:19 ntpd[18325]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:19.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:19 ntpd[18325]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-31T22:10:19.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:19 ntpd[18325]: can't open /var/log/ntpstats/peerstats.20240831: Permission denied 2024-08-31T22:10:19.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:19 ntpd[18274]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:19.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:19 ntpd[18274]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:19.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:19 ntpd[18274]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:19.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:19 ntpd[18274]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:19.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:19 ntpd[18274]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-31T22:10:19.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:19 ntpd[18274]: can't open /var/log/ntpstats/peerstats.20240831: Permission denied 2024-08-31T22:10:20.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:20 ntpd[18274]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:20.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:20 ntpd[18274]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:20.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:20 ntpd[18274]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:20.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:20 ntpd[18274]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:20.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:20 ntpd[18274]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-31T22:10:20.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:20 ntpd[18274]: can't open /var/log/ntpstats/peerstats.20240831: Permission denied 2024-08-31T22:10:21.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:21 ntpd[18325]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:21.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:21 ntpd[18325]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:21.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:21 ntpd[18325]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:21.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:21 ntpd[18325]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:21.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:21 ntpd[18325]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:21.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:21 ntpd[18325]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:21.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:21 ntpd[18325]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-31T22:10:21.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:21 ntpd[18325]: can't open /var/log/ntpstats/peerstats.20240831: Permission denied 2024-08-31T22:10:21.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:21 ntpd[18325]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:21.229 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:21 ntpd[18325]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:21.229 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:21 ntpd[18325]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-31T22:10:21.229 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:21 ntpd[18325]: can't open /var/log/ntpstats/peerstats.20240831: Permission denied 2024-08-31T22:10:21.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:21 ntpd[18274]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:21.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:21 ntpd[18274]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:21.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:21 ntpd[18274]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:21.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:21 ntpd[18274]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:21.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:21 ntpd[18274]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:21.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:21 ntpd[18274]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:21.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:21 ntpd[18274]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-31T22:10:21.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:21 ntpd[18274]: can't open /var/log/ntpstats/peerstats.20240831: Permission denied 2024-08-31T22:10:22.227 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:22 ntpd[18325]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:22.227 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:22 ntpd[18325]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:22.227 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:22 ntpd[18325]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:22 ntpd[18325]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:22 ntpd[18325]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:22 ntpd[18325]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:22 ntpd[18325]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-31T22:10:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:22 ntpd[18325]: can't open /var/log/ntpstats/peerstats.20240831: Permission denied 2024-08-31T22:10:22.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:22 ntpd[18274]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:22.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:22 ntpd[18274]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:22.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:22 ntpd[18274]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:22.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:22 ntpd[18274]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:22.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:22 ntpd[18274]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:22.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:22 ntpd[18274]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:22.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:22 ntpd[18274]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-31T22:10:22.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:22 ntpd[18274]: can't open /var/log/ntpstats/peerstats.20240831: Permission denied 2024-08-31T22:10:22.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:22 ntpd[18274]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:22.263 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:22 ntpd[18274]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:22.263 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:22 ntpd[18274]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-31T22:10:22.263 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:22 ntpd[18274]: can't open /var/log/ntpstats/peerstats.20240831: Permission denied 2024-08-31T22:10:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:23 ntpd[18325]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:23 ntpd[18325]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:23 ntpd[18325]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:23 ntpd[18325]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:23 ntpd[18325]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:23 ntpd[18325]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:23 ntpd[18325]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-31T22:10:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:23 ntpd[18325]: can't open /var/log/ntpstats/peerstats.20240831: Permission denied 2024-08-31T22:10:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:23 ntpd[18325]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:23 ntpd[18325]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:23 ntpd[18325]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-31T22:10:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:23 ntpd[18325]: can't open /var/log/ntpstats/peerstats.20240831: Permission denied 2024-08-31T22:10:23.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:23 ntpd[18274]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:23.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:23 ntpd[18274]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:23.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:23 ntpd[18274]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:23.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:23 ntpd[18274]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:23.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:23 ntpd[18274]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-31T22:10:23.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:23 ntpd[18274]: can't open /var/log/ntpstats/peerstats.20240831: Permission denied 2024-08-31T22:10:24.227 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:24 ntpd[18325]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:24.227 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:24 ntpd[18325]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:24.227 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:24 ntpd[18325]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:24.227 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:24 ntpd[18325]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:24.227 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:24 ntpd[18325]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-31T22:10:24.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:24 ntpd[18325]: can't open /var/log/ntpstats/peerstats.20240831: Permission denied 2024-08-31T22:10:24.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:24 ntpd[18274]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:24.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:24 ntpd[18274]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:24.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:24 ntpd[18274]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:24.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:24 ntpd[18274]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:24.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:24 ntpd[18274]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:24.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:24 ntpd[18274]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:24.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:24 ntpd[18274]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-31T22:10:24.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:24 ntpd[18274]: can't open /var/log/ntpstats/peerstats.20240831: Permission denied 2024-08-31T22:10:24.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:24 ntpd[18274]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:24.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:24 ntpd[18274]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:24.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:24 ntpd[18274]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-31T22:10:24.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:24 ntpd[18274]: can't open /var/log/ntpstats/peerstats.20240831: Permission denied 2024-08-31T22:10:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:25 ntpd[18325]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:25 ntpd[18325]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:25 ntpd[18325]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:25 ntpd[18325]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:25 ntpd[18325]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:25 ntpd[18325]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:25 ntpd[18325]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-31T22:10:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:25 ntpd[18325]: can't open /var/log/ntpstats/peerstats.20240831: Permission denied 2024-08-31T22:10:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:25 ntpd[18325]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:25 ntpd[18325]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:25 ntpd[18325]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-31T22:10:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:25 ntpd[18325]: can't open /var/log/ntpstats/peerstats.20240831: Permission denied 2024-08-31T22:10:25.229 INFO:teuthology.orchestra.run.smithi007.stdout:31 Aug 22:10:25 ntpd[18325]: ntpd: time slew -0.000696 s 2024-08-31T22:10:25.229 INFO:teuthology.orchestra.run.smithi007.stdout:ntpd: time slew -0.000696s 2024-08-31T22:10:25.230 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:25 ntpd[18325]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-08-31T22:10:25.230 INFO:teuthology.orchestra.run.smithi007.stderr:31 Aug 22:10:25 ntpd[18325]: can't open /var/log/ntpstats/loopstats.20240831: Permission denied 2024-08-31T22:10:25.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:25 ntpd[18274]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:25.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:25 ntpd[18274]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:25.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:25 ntpd[18274]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-31T22:10:25.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:25 ntpd[18274]: can't open /var/log/ntpstats/rawstats.20240831: Permission denied 2024-08-31T22:10:25.261 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:25 ntpd[18274]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-31T22:10:25.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:25 ntpd[18274]: can't open /var/log/ntpstats/peerstats.20240831: Permission denied 2024-08-31T22:10:25.262 INFO:teuthology.orchestra.run.smithi032.stdout:31 Aug 22:10:25 ntpd[18274]: ntpd: time slew -0.000961 s 2024-08-31T22:10:25.262 INFO:teuthology.orchestra.run.smithi032.stdout:ntpd: time slew -0.000961s 2024-08-31T22:10:25.262 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:25 ntpd[18274]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-08-31T22:10:25.263 INFO:teuthology.orchestra.run.smithi032.stderr:31 Aug 22:10:25 ntpd[18274]: can't open /var/log/ntpstats/loopstats.20240831: Permission denied 2024-08-31T22:10:25.272 INFO:teuthology.orchestra.run.smithi007.stdout: remote refid st t when poll reach delay offset jitter 2024-08-31T22:10:25.272 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================== 2024-08-31T22:10:25.272 INFO:teuthology.orchestra.run.smithi007.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-31T22:10:25.272 INFO:teuthology.orchestra.run.smithi007.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-31T22:10:25.272 INFO:teuthology.orchestra.run.smithi007.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-31T22:10:25.272 INFO:teuthology.orchestra.run.smithi007.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-31T22:10:25.312 INFO:teuthology.orchestra.run.smithi032.stdout: remote refid st t when poll reach delay offset jitter 2024-08-31T22:10:25.312 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================== 2024-08-31T22:10:25.312 INFO:teuthology.orchestra.run.smithi032.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-31T22:10:25.312 INFO:teuthology.orchestra.run.smithi032.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-31T22:10:25.312 INFO:teuthology.orchestra.run.smithi032.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-31T22:10:25.312 INFO:teuthology.orchestra.run.smithi032.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-31T22:10:25.313 INFO:teuthology.run_tasks:Running task install... 2024-08-31T22:10:25.320 DEBUG:teuthology.task.install:project ceph 2024-08-31T22:10:25.320 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3091dfc0187f68df2afd4e392dc4c381cfdc1cad'}} 2024-08-31T22:10:25.320 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '3091dfc0187f68df2afd4e392dc4c381cfdc1cad'} 2024-08-31T22:10:25.321 INFO:teuthology.task.install:Using flavor: default 2024-08-31T22:10:25.326 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-31T22:10:25.326 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-08-31T22:10:25.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-key list | grep Ceph 2024-08-31T22:10:25.327 DEBUG:teuthology.orchestra.run.smithi032:> sudo apt-key list | grep Ceph 2024-08-31T22:10:25.387 INFO:teuthology.orchestra.run.smithi007.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-08-31T22:10:25.413 INFO:teuthology.orchestra.run.smithi032.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-08-31T22:10:25.414 INFO:teuthology.orchestra.run.smithi007.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-08-31T22:10:25.414 INFO:teuthology.orchestra.run.smithi007.stdout:uid [ unknown] Ceph.com (release key) 2024-08-31T22:10:25.415 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2024-08-31T22:10:25.415 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3091dfc0187f68df2afd4e392dc4c381cfdc1cad 2024-08-31T22:10:25.450 INFO:teuthology.orchestra.run.smithi032.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-08-31T22:10:25.450 INFO:teuthology.orchestra.run.smithi032.stdout:uid [ unknown] Ceph.com (release key) 2024-08-31T22:10:25.450 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2024-08-31T22:10:25.451 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3091dfc0187f68df2afd4e392dc4c381cfdc1cad 2024-08-31T22:10:25.564 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default/ 2024-08-31T22:10:25.564 INFO:teuthology.task.install.deb:Package version is 19.1.1-244-g3091dfc0-1jammy 2024-08-31T22:10:25.600 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default/ 2024-08-31T22:10:25.600 INFO:teuthology.task.install.deb:Package version is 19.1.1-244-g3091dfc0-1jammy 2024-08-31T22:10:25.813 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:10:25.814 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-08-31T22:10:25.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get update 2024-08-31T22:10:25.848 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:10:25.848 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-08-31T22:10:25.862 DEBUG:teuthology.orchestra.run.smithi032:> sudo apt-get update 2024-08-31T22:10:26.132 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-08-31T22:10:26.132 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-08-31T22:10:26.164 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-08-31T22:10:26.164 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-08-31T22:10:26.170 INFO:teuthology.orchestra.run.smithi007.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy InRelease 2024-08-31T22:10:26.204 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-08-31T22:10:26.221 INFO:teuthology.orchestra.run.smithi032.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy InRelease 2024-08-31T22:10:26.221 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-08-31T22:10:26.236 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-08-31T22:10:26.281 INFO:teuthology.orchestra.run.smithi032.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-08-31T22:10:26.282 INFO:teuthology.orchestra.run.smithi007.stdout:Ign:6 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy Release.gpg 2024-08-31T22:10:26.285 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-08-31T22:10:26.317 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-08-31T22:10:26.343 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,417 B] 2024-08-31T22:10:26.343 INFO:teuthology.orchestra.run.smithi032.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy Release.gpg 2024-08-31T22:10:26.404 INFO:teuthology.orchestra.run.smithi032.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,417 B] 2024-08-31T22:10:26.409 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-08-31T22:10:26.474 INFO:teuthology.orchestra.run.smithi032.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-08-31T22:10:26.547 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 29.0 kB in 1s (54.5 kB/s) 2024-08-31T22:10:26.554 INFO:teuthology.orchestra.run.smithi032.stdout:Fetched 29.0 kB in 1s (57.3 kB/s) 2024-08-31T22:10:27.551 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2024-08-31T22:10:27.566 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-244-g3091dfc0-1jammy cephadm=19.1.1-244-g3091dfc0-1jammy ceph-mds=19.1.1-244-g3091dfc0-1jammy ceph-mgr=19.1.1-244-g3091dfc0-1jammy ceph-common=19.1.1-244-g3091dfc0-1jammy ceph-fuse=19.1.1-244-g3091dfc0-1jammy ceph-test=19.1.1-244-g3091dfc0-1jammy ceph-volume=19.1.1-244-g3091dfc0-1jammy radosgw=19.1.1-244-g3091dfc0-1jammy python3-rados=19.1.1-244-g3091dfc0-1jammy python3-rgw=19.1.1-244-g3091dfc0-1jammy python3-cephfs=19.1.1-244-g3091dfc0-1jammy python3-rbd=19.1.1-244-g3091dfc0-1jammy libcephfs2=19.1.1-244-g3091dfc0-1jammy libcephfs-dev=19.1.1-244-g3091dfc0-1jammy librados2=19.1.1-244-g3091dfc0-1jammy librbd1=19.1.1-244-g3091dfc0-1jammy rbd-fuse=19.1.1-244-g3091dfc0-1jammy rbd-mirror=19.1.1-244-g3091dfc0-1jammy 2024-08-31T22:10:27.604 INFO:teuthology.orchestra.run.smithi032.stdout:Reading package lists... 2024-08-31T22:10:27.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-244-g3091dfc0-1jammy cephadm=19.1.1-244-g3091dfc0-1jammy ceph-mds=19.1.1-244-g3091dfc0-1jammy ceph-mgr=19.1.1-244-g3091dfc0-1jammy ceph-common=19.1.1-244-g3091dfc0-1jammy ceph-fuse=19.1.1-244-g3091dfc0-1jammy ceph-test=19.1.1-244-g3091dfc0-1jammy ceph-volume=19.1.1-244-g3091dfc0-1jammy radosgw=19.1.1-244-g3091dfc0-1jammy python3-rados=19.1.1-244-g3091dfc0-1jammy python3-rgw=19.1.1-244-g3091dfc0-1jammy python3-cephfs=19.1.1-244-g3091dfc0-1jammy python3-rbd=19.1.1-244-g3091dfc0-1jammy libcephfs2=19.1.1-244-g3091dfc0-1jammy libcephfs-dev=19.1.1-244-g3091dfc0-1jammy librados2=19.1.1-244-g3091dfc0-1jammy librbd1=19.1.1-244-g3091dfc0-1jammy rbd-fuse=19.1.1-244-g3091dfc0-1jammy rbd-mirror=19.1.1-244-g3091dfc0-1jammy 2024-08-31T22:10:27.621 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2024-08-31T22:10:27.674 INFO:teuthology.orchestra.run.smithi032.stdout:Reading package lists... 2024-08-31T22:10:27.810 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2024-08-31T22:10:27.810 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2024-08-31T22:10:27.863 INFO:teuthology.orchestra.run.smithi032.stdout:Building dependency tree... 2024-08-31T22:10:27.863 INFO:teuthology.orchestra.run.smithi032.stdout:Reading state information... 2024-08-31T22:10:28.001 INFO:teuthology.orchestra.run.smithi007.stdout:The following packages were automatically installed and are no longer required: 2024-08-31T22:10:28.001 INFO:teuthology.orchestra.run.smithi007.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-08-31T22:10:28.001 INFO:teuthology.orchestra.run.smithi007.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-08-31T22:10:28.001 INFO:teuthology.orchestra.run.smithi007.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-08-31T22:10:28.001 INFO:teuthology.orchestra.run.smithi007.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-08-31T22:10:28.002 INFO:teuthology.orchestra.run.smithi007.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-08-31T22:10:28.002 INFO:teuthology.orchestra.run.smithi007.stdout:Use 'sudo apt autoremove' to remove them. 2024-08-31T22:10:28.002 INFO:teuthology.orchestra.run.smithi007.stdout:The following additional packages will be installed: 2024-08-31T22:10:28.003 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-08-31T22:10:28.003 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-08-31T22:10:28.003 INFO:teuthology.orchestra.run.smithi007.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-08-31T22:10:28.003 INFO:teuthology.orchestra.run.smithi007.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-08-31T22:10:28.004 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-08-31T22:10:28.004 INFO:teuthology.orchestra.run.smithi007.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-08-31T22:10:28.004 INFO:teuthology.orchestra.run.smithi007.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-08-31T22:10:28.004 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-08-31T22:10:28.004 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-08-31T22:10:28.004 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-08-31T22:10:28.004 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-08-31T22:10:28.004 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-08-31T22:10:28.005 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-08-31T22:10:28.005 INFO:teuthology.orchestra.run.smithi007.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-08-31T22:10:28.005 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-08-31T22:10:28.005 INFO:teuthology.orchestra.run.smithi007.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-08-31T22:10:28.005 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-08-31T22:10:28.005 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-08-31T22:10:28.005 INFO:teuthology.orchestra.run.smithi007.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-08-31T22:10:28.005 INFO:teuthology.orchestra.run.smithi007.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-08-31T22:10:28.005 INFO:teuthology.orchestra.run.smithi007.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-08-31T22:10:28.005 INFO:teuthology.orchestra.run.smithi007.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-08-31T22:10:28.005 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-08-31T22:10:28.005 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet zip 2024-08-31T22:10:28.006 INFO:teuthology.orchestra.run.smithi007.stdout:Suggested packages: 2024-08-31T22:10:28.006 INFO:teuthology.orchestra.run.smithi007.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-08-31T22:10:28.006 INFO:teuthology.orchestra.run.smithi007.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-08-31T22:10:28.006 INFO:teuthology.orchestra.run.smithi007.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-08-31T22:10:28.006 INFO:teuthology.orchestra.run.smithi007.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-08-31T22:10:28.006 INFO:teuthology.orchestra.run.smithi007.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-08-31T22:10:28.006 INFO:teuthology.orchestra.run.smithi007.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-08-31T22:10:28.007 INFO:teuthology.orchestra.run.smithi007.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-08-31T22:10:28.007 INFO:teuthology.orchestra.run.smithi007.stdout:Recommended packages: 2024-08-31T22:10:28.007 INFO:teuthology.orchestra.run.smithi007.stdout: btrfs-tools 2024-08-31T22:10:28.055 INFO:teuthology.orchestra.run.smithi032.stdout:The following packages were automatically installed and are no longer required: 2024-08-31T22:10:28.055 INFO:teuthology.orchestra.run.smithi032.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-08-31T22:10:28.055 INFO:teuthology.orchestra.run.smithi032.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-08-31T22:10:28.055 INFO:teuthology.orchestra.run.smithi032.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-08-31T22:10:28.056 INFO:teuthology.orchestra.run.smithi032.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-08-31T22:10:28.056 INFO:teuthology.orchestra.run.smithi032.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-08-31T22:10:28.056 INFO:teuthology.orchestra.run.smithi032.stdout:Use 'sudo apt autoremove' to remove them. 2024-08-31T22:10:28.057 INFO:teuthology.orchestra.run.smithi032.stdout:The following additional packages will be installed: 2024-08-31T22:10:28.058 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-08-31T22:10:28.058 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-08-31T22:10:28.058 INFO:teuthology.orchestra.run.smithi032.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-08-31T22:10:28.058 INFO:teuthology.orchestra.run.smithi032.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-08-31T22:10:28.058 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-08-31T22:10:28.058 INFO:teuthology.orchestra.run.smithi032.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-08-31T22:10:28.058 INFO:teuthology.orchestra.run.smithi032.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-08-31T22:10:28.058 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-08-31T22:10:28.058 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-08-31T22:10:28.058 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-08-31T22:10:28.058 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-08-31T22:10:28.058 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-08-31T22:10:28.059 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-08-31T22:10:28.059 INFO:teuthology.orchestra.run.smithi032.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-08-31T22:10:28.059 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-08-31T22:10:28.059 INFO:teuthology.orchestra.run.smithi032.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-08-31T22:10:28.059 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-08-31T22:10:28.060 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-08-31T22:10:28.060 INFO:teuthology.orchestra.run.smithi032.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-08-31T22:10:28.060 INFO:teuthology.orchestra.run.smithi032.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-08-31T22:10:28.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-08-31T22:10:28.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-08-31T22:10:28.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-08-31T22:10:28.061 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet zip 2024-08-31T22:10:28.061 INFO:teuthology.orchestra.run.smithi032.stdout:Suggested packages: 2024-08-31T22:10:28.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-08-31T22:10:28.061 INFO:teuthology.orchestra.run.smithi032.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-08-31T22:10:28.062 INFO:teuthology.orchestra.run.smithi032.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-08-31T22:10:28.062 INFO:teuthology.orchestra.run.smithi032.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-08-31T22:10:28.062 INFO:teuthology.orchestra.run.smithi032.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-08-31T22:10:28.062 INFO:teuthology.orchestra.run.smithi032.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-08-31T22:10:28.062 INFO:teuthology.orchestra.run.smithi032.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-08-31T22:10:28.062 INFO:teuthology.orchestra.run.smithi032.stdout:Recommended packages: 2024-08-31T22:10:28.062 INFO:teuthology.orchestra.run.smithi032.stdout: btrfs-tools 2024-08-31T22:10:28.112 INFO:teuthology.orchestra.run.smithi007.stdout:The following NEW packages will be installed: 2024-08-31T22:10:28.112 INFO:teuthology.orchestra.run.smithi007.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-08-31T22:10:28.112 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-08-31T22:10:28.112 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-08-31T22:10:28.112 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-08-31T22:10:28.112 INFO:teuthology.orchestra.run.smithi007.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-08-31T22:10:28.112 INFO:teuthology.orchestra.run.smithi007.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-08-31T22:10:28.113 INFO:teuthology.orchestra.run.smithi007.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-08-31T22:10:28.113 INFO:teuthology.orchestra.run.smithi007.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-08-31T22:10:28.113 INFO:teuthology.orchestra.run.smithi007.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-08-31T22:10:28.113 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-08-31T22:10:28.113 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-08-31T22:10:28.113 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-08-31T22:10:28.113 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-08-31T22:10:28.113 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-08-31T22:10:28.113 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-08-31T22:10:28.113 INFO:teuthology.orchestra.run.smithi007.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-08-31T22:10:28.113 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-08-31T22:10:28.113 INFO:teuthology.orchestra.run.smithi007.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-08-31T22:10:28.113 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-08-31T22:10:28.114 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-08-31T22:10:28.114 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-08-31T22:10:28.114 INFO:teuthology.orchestra.run.smithi007.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-08-31T22:10:28.114 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-08-31T22:10:28.114 INFO:teuthology.orchestra.run.smithi007.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-08-31T22:10:28.114 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-08-31T22:10:28.114 INFO:teuthology.orchestra.run.smithi007.stdout: radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip 2024-08-31T22:10:28.165 INFO:teuthology.orchestra.run.smithi032.stdout:The following NEW packages will be installed: 2024-08-31T22:10:28.165 INFO:teuthology.orchestra.run.smithi032.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-08-31T22:10:28.165 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-08-31T22:10:28.165 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-08-31T22:10:28.165 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-08-31T22:10:28.165 INFO:teuthology.orchestra.run.smithi032.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-08-31T22:10:28.165 INFO:teuthology.orchestra.run.smithi032.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-08-31T22:10:28.166 INFO:teuthology.orchestra.run.smithi032.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-08-31T22:10:28.166 INFO:teuthology.orchestra.run.smithi032.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-08-31T22:10:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-08-31T22:10:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-08-31T22:10:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-08-31T22:10:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-08-31T22:10:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-08-31T22:10:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-08-31T22:10:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-08-31T22:10:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-08-31T22:10:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-08-31T22:10:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-08-31T22:10:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-08-31T22:10:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-08-31T22:10:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-08-31T22:10:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-08-31T22:10:28.168 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-08-31T22:10:28.168 INFO:teuthology.orchestra.run.smithi032.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-08-31T22:10:28.168 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-08-31T22:10:28.168 INFO:teuthology.orchestra.run.smithi032.stdout: radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip 2024-08-31T22:10:28.303 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-08-31T22:10:28.303 INFO:teuthology.orchestra.run.smithi007.stdout:Need to get 184 MB of archives. 2024-08-31T22:10:28.303 INFO:teuthology.orchestra.run.smithi007.stdout:After this operation, 842 MB of additional disk space will be used. 2024-08-31T22:10:28.303 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-08-31T22:10:28.356 INFO:teuthology.orchestra.run.smithi032.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-08-31T22:10:28.356 INFO:teuthology.orchestra.run.smithi032.stdout:Need to get 184 MB of archives. 2024-08-31T22:10:28.356 INFO:teuthology.orchestra.run.smithi032.stdout:After this operation, 842 MB of additional disk space will be used. 2024-08-31T22:10:28.356 INFO:teuthology.orchestra.run.smithi032.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-08-31T22:10:28.512 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-244-g3091dfc0-1jammy [3,589 kB] 2024-08-31T22:10:28.562 INFO:teuthology.orchestra.run.smithi032.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-244-g3091dfc0-1jammy [3,589 kB] 2024-08-31T22:10:28.779 INFO:teuthology.orchestra.run.smithi032.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-08-31T22:10:28.792 INFO:teuthology.orchestra.run.smithi032.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-08-31T22:10:28.873 INFO:teuthology.orchestra.run.smithi032.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-08-31T22:10:28.951 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-08-31T22:10:28.999 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-08-31T22:10:29.068 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-08-31T22:10:29.090 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-244-g3091dfc0-1jammy [3,240 kB] 2024-08-31T22:10:29.101 INFO:teuthology.orchestra.run.smithi032.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-08-31T22:10:29.112 INFO:teuthology.orchestra.run.smithi032.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-08-31T22:10:29.142 INFO:teuthology.orchestra.run.smithi032.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-08-31T22:10:29.150 INFO:teuthology.orchestra.run.smithi032.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-08-31T22:10:29.151 INFO:teuthology.orchestra.run.smithi032.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-08-31T22:10:29.153 INFO:teuthology.orchestra.run.smithi032.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-08-31T22:10:29.155 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-08-31T22:10:29.171 INFO:teuthology.orchestra.run.smithi032.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-08-31T22:10:29.175 INFO:teuthology.orchestra.run.smithi032.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-08-31T22:10:29.178 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-08-31T22:10:29.180 INFO:teuthology.orchestra.run.smithi032.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-08-31T22:10:29.183 INFO:teuthology.orchestra.run.smithi032.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-08-31T22:10:29.212 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-08-31T22:10:29.235 INFO:teuthology.orchestra.run.smithi007.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-08-31T22:10:29.253 INFO:teuthology.orchestra.run.smithi007.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-08-31T22:10:29.264 INFO:teuthology.orchestra.run.smithi032.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-08-31T22:10:29.269 INFO:teuthology.orchestra.run.smithi032.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-08-31T22:10:29.271 INFO:teuthology.orchestra.run.smithi007.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-08-31T22:10:29.273 INFO:teuthology.orchestra.run.smithi032.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-08-31T22:10:29.276 INFO:teuthology.orchestra.run.smithi032.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-08-31T22:10:29.277 INFO:teuthology.orchestra.run.smithi032.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-08-31T22:10:29.278 INFO:teuthology.orchestra.run.smithi032.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-08-31T22:10:29.280 INFO:teuthology.orchestra.run.smithi032.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-08-31T22:10:29.280 INFO:teuthology.orchestra.run.smithi032.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-08-31T22:10:29.281 INFO:teuthology.orchestra.run.smithi032.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-08-31T22:10:29.303 INFO:teuthology.orchestra.run.smithi007.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-08-31T22:10:29.325 INFO:teuthology.orchestra.run.smithi007.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-08-31T22:10:29.346 INFO:teuthology.orchestra.run.smithi032.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-08-31T22:10:29.346 INFO:teuthology.orchestra.run.smithi007.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-08-31T22:10:29.348 INFO:teuthology.orchestra.run.smithi007.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-244-g3091dfc0-1jammy [719 kB] 2024-08-31T22:10:29.364 INFO:teuthology.orchestra.run.smithi032.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-08-31T22:10:29.364 INFO:teuthology.orchestra.run.smithi032.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-08-31T22:10:29.364 INFO:teuthology.orchestra.run.smithi032.stdout:Get:28 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-244-g3091dfc0-1jammy [3,240 kB] 2024-08-31T22:10:29.365 INFO:teuthology.orchestra.run.smithi032.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-08-31T22:10:29.365 INFO:teuthology.orchestra.run.smithi007.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-08-31T22:10:29.370 INFO:teuthology.orchestra.run.smithi032.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-08-31T22:10:29.371 INFO:teuthology.orchestra.run.smithi032.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-08-31T22:10:29.383 INFO:teuthology.orchestra.run.smithi007.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-08-31T22:10:29.400 INFO:teuthology.orchestra.run.smithi007.stdout:Get:19 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-244-g3091dfc0-1jammy [357 kB] 2024-08-31T22:10:29.405 INFO:teuthology.orchestra.run.smithi007.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-08-31T22:10:29.427 INFO:teuthology.orchestra.run.smithi007.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-08-31T22:10:29.428 INFO:teuthology.orchestra.run.smithi032.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-08-31T22:10:29.428 INFO:teuthology.orchestra.run.smithi032.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-08-31T22:10:29.430 INFO:teuthology.orchestra.run.smithi032.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-08-31T22:10:29.431 INFO:teuthology.orchestra.run.smithi032.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-08-31T22:10:29.440 INFO:teuthology.orchestra.run.smithi007.stdout:Get:22 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-244-g3091dfc0-1jammy [32.8 kB] 2024-08-31T22:10:29.441 INFO:teuthology.orchestra.run.smithi007.stdout:Get:23 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-244-g3091dfc0-1jammy [180 kB] 2024-08-31T22:10:29.446 INFO:teuthology.orchestra.run.smithi007.stdout:Get:24 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-244-g3091dfc0-1jammy [67.6 kB] 2024-08-31T22:10:29.447 INFO:teuthology.orchestra.run.smithi007.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-08-31T22:10:29.448 INFO:teuthology.orchestra.run.smithi007.stdout:Get:26 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-244-g3091dfc0-1jammy [333 kB] 2024-08-31T22:10:29.466 INFO:teuthology.orchestra.run.smithi007.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-08-31T22:10:29.483 INFO:teuthology.orchestra.run.smithi007.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-08-31T22:10:29.488 INFO:teuthology.orchestra.run.smithi007.stdout:Get:29 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-244-g3091dfc0-1jammy [6,874 kB] 2024-08-31T22:10:29.503 INFO:teuthology.orchestra.run.smithi007.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-08-31T22:10:29.508 INFO:teuthology.orchestra.run.smithi032.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-08-31T22:10:29.511 INFO:teuthology.orchestra.run.smithi032.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-08-31T22:10:29.513 INFO:teuthology.orchestra.run.smithi032.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-08-31T22:10:29.514 INFO:teuthology.orchestra.run.smithi032.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-08-31T22:10:29.520 INFO:teuthology.orchestra.run.smithi007.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-08-31T22:10:29.524 INFO:teuthology.orchestra.run.smithi032.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-08-31T22:10:29.539 INFO:teuthology.orchestra.run.smithi007.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-08-31T22:10:29.556 INFO:teuthology.orchestra.run.smithi007.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-08-31T22:10:29.590 INFO:teuthology.orchestra.run.smithi032.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-08-31T22:10:29.591 INFO:teuthology.orchestra.run.smithi032.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-08-31T22:10:29.592 INFO:teuthology.orchestra.run.smithi032.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-08-31T22:10:29.595 INFO:teuthology.orchestra.run.smithi032.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-08-31T22:10:29.598 INFO:teuthology.orchestra.run.smithi032.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-08-31T22:10:29.635 INFO:teuthology.orchestra.run.smithi007.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-08-31T22:10:29.649 INFO:teuthology.orchestra.run.smithi032.stdout:Get:46 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-244-g3091dfc0-1jammy [719 kB] 2024-08-31T22:10:29.652 INFO:teuthology.orchestra.run.smithi007.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-08-31T22:10:29.670 INFO:teuthology.orchestra.run.smithi007.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-08-31T22:10:29.671 INFO:teuthology.orchestra.run.smithi032.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-08-31T22:10:29.674 INFO:teuthology.orchestra.run.smithi032.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-08-31T22:10:29.675 INFO:teuthology.orchestra.run.smithi032.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-08-31T22:10:29.680 INFO:teuthology.orchestra.run.smithi032.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-08-31T22:10:29.681 INFO:teuthology.orchestra.run.smithi032.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-08-31T22:10:29.692 INFO:teuthology.orchestra.run.smithi032.stdout:Get:52 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-244-g3091dfc0-1jammy [357 kB] 2024-08-31T22:10:29.692 INFO:teuthology.orchestra.run.smithi007.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-08-31T22:10:29.709 INFO:teuthology.orchestra.run.smithi007.stdout:Get:38 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-244-g3091dfc0-1jammy [112 kB] 2024-08-31T22:10:29.711 INFO:teuthology.orchestra.run.smithi007.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-08-31T22:10:29.713 INFO:teuthology.orchestra.run.smithi032.stdout:Get:53 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-244-g3091dfc0-1jammy [32.8 kB] 2024-08-31T22:10:29.714 INFO:teuthology.orchestra.run.smithi032.stdout:Get:54 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-244-g3091dfc0-1jammy [180 kB] 2024-08-31T22:10:29.715 INFO:teuthology.orchestra.run.smithi007.stdout:Get:40 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-244-g3091dfc0-1jammy [472 kB] 2024-08-31T22:10:29.727 INFO:teuthology.orchestra.run.smithi007.stdout:Get:41 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-244-g3091dfc0-1jammy [25.9 MB] 2024-08-31T22:10:29.729 INFO:teuthology.orchestra.run.smithi007.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-08-31T22:10:29.730 INFO:teuthology.orchestra.run.smithi032.stdout:Get:55 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-244-g3091dfc0-1jammy [67.6 kB] 2024-08-31T22:10:29.732 INFO:teuthology.orchestra.run.smithi032.stdout:Get:56 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-244-g3091dfc0-1jammy [333 kB] 2024-08-31T22:10:29.747 INFO:teuthology.orchestra.run.smithi007.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-08-31T22:10:29.752 INFO:teuthology.orchestra.run.smithi032.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-08-31T22:10:29.753 INFO:teuthology.orchestra.run.smithi032.stdout:Get:58 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-244-g3091dfc0-1jammy [6,874 kB] 2024-08-31T22:10:29.757 INFO:teuthology.orchestra.run.smithi032.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-08-31T22:10:29.766 INFO:teuthology.orchestra.run.smithi007.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-08-31T22:10:29.784 INFO:teuthology.orchestra.run.smithi007.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-08-31T22:10:29.802 INFO:teuthology.orchestra.run.smithi007.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-08-31T22:10:29.821 INFO:teuthology.orchestra.run.smithi007.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-08-31T22:10:29.840 INFO:teuthology.orchestra.run.smithi007.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-08-31T22:10:29.858 INFO:teuthology.orchestra.run.smithi007.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-08-31T22:10:29.886 INFO:teuthology.orchestra.run.smithi007.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-08-31T22:10:29.903 INFO:teuthology.orchestra.run.smithi007.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-08-31T22:10:29.921 INFO:teuthology.orchestra.run.smithi007.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-08-31T22:10:29.923 INFO:teuthology.orchestra.run.smithi032.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-08-31T22:10:29.925 INFO:teuthology.orchestra.run.smithi032.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-08-31T22:10:29.928 INFO:teuthology.orchestra.run.smithi032.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-08-31T22:10:29.929 INFO:teuthology.orchestra.run.smithi032.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-08-31T22:10:29.931 INFO:teuthology.orchestra.run.smithi032.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-08-31T22:10:29.940 INFO:teuthology.orchestra.run.smithi007.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-08-31T22:10:29.959 INFO:teuthology.orchestra.run.smithi007.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-08-31T22:10:29.969 INFO:teuthology.orchestra.run.smithi032.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-08-31T22:10:29.992 INFO:teuthology.orchestra.run.smithi007.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-08-31T22:10:29.997 INFO:teuthology.orchestra.run.smithi032.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-08-31T22:10:29.998 INFO:teuthology.orchestra.run.smithi032.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-08-31T22:10:30.011 INFO:teuthology.orchestra.run.smithi007.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-08-31T22:10:30.028 INFO:teuthology.orchestra.run.smithi032.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-08-31T22:10:30.028 INFO:teuthology.orchestra.run.smithi032.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-08-31T22:10:30.031 INFO:teuthology.orchestra.run.smithi007.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-08-31T22:10:30.050 INFO:teuthology.orchestra.run.smithi007.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-08-31T22:10:30.071 INFO:teuthology.orchestra.run.smithi007.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-08-31T22:10:30.089 INFO:teuthology.orchestra.run.smithi007.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-08-31T22:10:30.090 INFO:teuthology.orchestra.run.smithi032.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-08-31T22:10:30.091 INFO:teuthology.orchestra.run.smithi032.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-08-31T22:10:30.092 INFO:teuthology.orchestra.run.smithi032.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-08-31T22:10:30.093 INFO:teuthology.orchestra.run.smithi032.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-08-31T22:10:30.094 INFO:teuthology.orchestra.run.smithi032.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-08-31T22:10:30.095 INFO:teuthology.orchestra.run.smithi032.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-08-31T22:10:30.102 INFO:teuthology.orchestra.run.smithi032.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-08-31T22:10:30.171 INFO:teuthology.orchestra.run.smithi007.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-08-31T22:10:30.172 INFO:teuthology.orchestra.run.smithi032.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-08-31T22:10:30.175 INFO:teuthology.orchestra.run.smithi032.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-08-31T22:10:30.177 INFO:teuthology.orchestra.run.smithi032.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-08-31T22:10:30.185 INFO:teuthology.orchestra.run.smithi032.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-244-g3091dfc0-1jammy [112 kB] 2024-08-31T22:10:30.188 INFO:teuthology.orchestra.run.smithi032.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-244-g3091dfc0-1jammy [472 kB] 2024-08-31T22:10:30.192 INFO:teuthology.orchestra.run.smithi007.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-08-31T22:10:30.223 INFO:teuthology.orchestra.run.smithi032.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-244-g3091dfc0-1jammy [25.9 MB] 2024-08-31T22:10:30.253 INFO:teuthology.orchestra.run.smithi032.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-08-31T22:10:30.260 INFO:teuthology.orchestra.run.smithi032.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-08-31T22:10:30.262 INFO:teuthology.orchestra.run.smithi032.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-08-31T22:10:30.264 INFO:teuthology.orchestra.run.smithi032.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-08-31T22:10:30.275 INFO:teuthology.orchestra.run.smithi032.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-08-31T22:10:30.276 INFO:teuthology.orchestra.run.smithi032.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-08-31T22:10:30.284 INFO:teuthology.orchestra.run.smithi032.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-08-31T22:10:30.297 INFO:teuthology.orchestra.run.smithi007.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-08-31T22:10:30.316 INFO:teuthology.orchestra.run.smithi007.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-08-31T22:10:30.334 INFO:teuthology.orchestra.run.smithi032.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-08-31T22:10:30.335 INFO:teuthology.orchestra.run.smithi007.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-08-31T22:10:30.336 INFO:teuthology.orchestra.run.smithi032.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-08-31T22:10:30.336 INFO:teuthology.orchestra.run.smithi032.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-08-31T22:10:30.352 INFO:teuthology.orchestra.run.smithi007.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-08-31T22:10:30.371 INFO:teuthology.orchestra.run.smithi007.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-08-31T22:10:30.415 INFO:teuthology.orchestra.run.smithi032.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-08-31T22:10:30.417 INFO:teuthology.orchestra.run.smithi032.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-08-31T22:10:30.433 INFO:teuthology.orchestra.run.smithi007.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-08-31T22:10:30.437 INFO:teuthology.orchestra.run.smithi032.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-08-31T22:10:30.438 INFO:teuthology.orchestra.run.smithi032.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-08-31T22:10:30.439 INFO:teuthology.orchestra.run.smithi032.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-08-31T22:10:30.444 INFO:teuthology.orchestra.run.smithi032.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-08-31T22:10:30.446 INFO:teuthology.orchestra.run.smithi032.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-08-31T22:10:30.455 INFO:teuthology.orchestra.run.smithi007.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-08-31T22:10:30.473 INFO:teuthology.orchestra.run.smithi007.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-08-31T22:10:30.531 INFO:teuthology.orchestra.run.smithi007.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-08-31T22:10:30.549 INFO:teuthology.orchestra.run.smithi007.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-08-31T22:10:30.601 INFO:teuthology.orchestra.run.smithi007.stdout:Get:73 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-244-g3091dfc0-1jammy [5,144 kB] 2024-08-31T22:10:30.616 INFO:teuthology.orchestra.run.smithi007.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-08-31T22:10:30.634 INFO:teuthology.orchestra.run.smithi007.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-08-31T22:10:30.652 INFO:teuthology.orchestra.run.smithi007.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-08-31T22:10:30.671 INFO:teuthology.orchestra.run.smithi007.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-08-31T22:10:30.689 INFO:teuthology.orchestra.run.smithi007.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-08-31T22:10:30.708 INFO:teuthology.orchestra.run.smithi007.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-08-31T22:10:30.730 INFO:teuthology.orchestra.run.smithi007.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-244-g3091dfc0-1jammy [245 kB] 2024-08-31T22:10:30.733 INFO:teuthology.orchestra.run.smithi007.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-08-31T22:10:30.735 INFO:teuthology.orchestra.run.smithi007.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-244-g3091dfc0-1jammy [124 kB] 2024-08-31T22:10:30.738 INFO:teuthology.orchestra.run.smithi007.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-244-g3091dfc0-1jammy [1,555 kB] 2024-08-31T22:10:30.754 INFO:teuthology.orchestra.run.smithi007.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-08-31T22:10:30.775 INFO:teuthology.orchestra.run.smithi007.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-08-31T22:10:30.781 INFO:teuthology.orchestra.run.smithi007.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-244-g3091dfc0-1jammy [6,142 kB] 2024-08-31T22:10:30.794 INFO:teuthology.orchestra.run.smithi007.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-08-31T22:10:30.819 INFO:teuthology.orchestra.run.smithi007.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-08-31T22:10:30.843 INFO:teuthology.orchestra.run.smithi007.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-08-31T22:10:30.927 INFO:teuthology.orchestra.run.smithi007.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-08-31T22:10:30.945 INFO:teuthology.orchestra.run.smithi007.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-08-31T22:10:30.963 INFO:teuthology.orchestra.run.smithi007.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-244-g3091dfc0-1jammy [22.8 MB] 2024-08-31T22:10:30.973 INFO:teuthology.orchestra.run.smithi007.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-08-31T22:10:30.992 INFO:teuthology.orchestra.run.smithi007.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-08-31T22:10:31.016 INFO:teuthology.orchestra.run.smithi007.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-08-31T22:10:31.034 INFO:teuthology.orchestra.run.smithi007.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-08-31T22:10:31.053 INFO:teuthology.orchestra.run.smithi007.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-08-31T22:10:31.071 INFO:teuthology.orchestra.run.smithi007.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-08-31T22:10:31.092 INFO:teuthology.orchestra.run.smithi007.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-08-31T22:10:31.112 INFO:teuthology.orchestra.run.smithi007.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-08-31T22:10:31.146 INFO:teuthology.orchestra.run.smithi007.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-08-31T22:10:31.164 INFO:teuthology.orchestra.run.smithi007.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-08-31T22:10:31.182 INFO:teuthology.orchestra.run.smithi007.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-08-31T22:10:31.204 INFO:teuthology.orchestra.run.smithi007.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-08-31T22:10:31.223 INFO:teuthology.orchestra.run.smithi007.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-08-31T22:10:31.697 INFO:teuthology.orchestra.run.smithi007.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-244-g3091dfc0-1jammy [14.1 kB] 2024-08-31T22:10:31.698 INFO:teuthology.orchestra.run.smithi007.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-244-g3091dfc0-1jammy [877 kB] 2024-08-31T22:10:31.714 INFO:teuthology.orchestra.run.smithi007.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-244-g3091dfc0-1jammy [2,462 kB] 2024-08-31T22:10:31.765 INFO:teuthology.orchestra.run.smithi007.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-244-g3091dfc0-1jammy [794 kB] 2024-08-31T22:10:31.785 INFO:teuthology.orchestra.run.smithi007.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-244-g3091dfc0-1jammy [152 kB] 2024-08-31T22:10:31.788 INFO:teuthology.orchestra.run.smithi007.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-244-g3091dfc0-1jammy [2,323 kB] 2024-08-31T22:10:31.819 INFO:teuthology.orchestra.run.smithi032.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-244-g3091dfc0-1jammy [5,144 kB] 2024-08-31T22:10:31.851 INFO:teuthology.orchestra.run.smithi007.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-244-g3091dfc0-1jammy [8,625 kB] 2024-08-31T22:10:32.083 INFO:teuthology.orchestra.run.smithi032.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-244-g3091dfc0-1jammy [245 kB] 2024-08-31T22:10:32.088 INFO:teuthology.orchestra.run.smithi032.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-244-g3091dfc0-1jammy [124 kB] 2024-08-31T22:10:32.101 INFO:teuthology.orchestra.run.smithi007.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-244-g3091dfc0-1jammy [14.2 kB] 2024-08-31T22:10:32.101 INFO:teuthology.orchestra.run.smithi007.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-244-g3091dfc0-1jammy [51.3 MB] 2024-08-31T22:10:32.103 INFO:teuthology.orchestra.run.smithi032.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-244-g3091dfc0-1jammy [1,555 kB] 2024-08-31T22:10:32.195 INFO:teuthology.orchestra.run.smithi032.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-244-g3091dfc0-1jammy [6,142 kB] 2024-08-31T22:10:32.561 INFO:teuthology.orchestra.run.smithi032.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-244-g3091dfc0-1jammy [22.8 MB] 2024-08-31T22:10:33.901 INFO:teuthology.orchestra.run.smithi007.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-244-g3091dfc0-1jammy [130 kB] 2024-08-31T22:10:33.904 INFO:teuthology.orchestra.run.smithi007.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-244-g3091dfc0-1jammy [39.0 kB] 2024-08-31T22:10:33.905 INFO:teuthology.orchestra.run.smithi007.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-244-g3091dfc0-1jammy [13.6 MB] 2024-08-31T22:10:33.954 INFO:teuthology.orchestra.run.smithi032.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-244-g3091dfc0-1jammy [14.1 kB] 2024-08-31T22:10:33.955 INFO:teuthology.orchestra.run.smithi032.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-244-g3091dfc0-1jammy [877 kB] 2024-08-31T22:10:33.972 INFO:teuthology.orchestra.run.smithi032.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-244-g3091dfc0-1jammy [2,462 kB] 2024-08-31T22:10:34.085 INFO:teuthology.orchestra.run.smithi032.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-244-g3091dfc0-1jammy [794 kB] 2024-08-31T22:10:34.127 INFO:teuthology.orchestra.run.smithi032.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-244-g3091dfc0-1jammy [152 kB] 2024-08-31T22:10:34.130 INFO:teuthology.orchestra.run.smithi032.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-244-g3091dfc0-1jammy [2,323 kB] 2024-08-31T22:10:34.240 INFO:teuthology.orchestra.run.smithi032.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-244-g3091dfc0-1jammy [8,625 kB] 2024-08-31T22:10:34.297 INFO:teuthology.orchestra.run.smithi007.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-244-g3091dfc0-1jammy [91.4 kB] 2024-08-31T22:10:34.299 INFO:teuthology.orchestra.run.smithi007.stdout:Get:119 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.1.1-244-g3091dfc0-1jammy [3,140 kB] 2024-08-31T22:10:34.651 INFO:teuthology.orchestra.run.smithi032.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-244-g3091dfc0-1jammy [14.2 kB] 2024-08-31T22:10:34.652 INFO:teuthology.orchestra.run.smithi032.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-244-g3091dfc0-1jammy [51.3 MB] 2024-08-31T22:10:35.022 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 184 MB in 6s (29.2 MB/s) 2024-08-31T22:10:35.186 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-08-31T22:10:35.289 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-08-31T22:10:35.295 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-08-31T22:10:35.368 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-08-31T22:10:35.680 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-08-31T22:10:35.707 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-08-31T22:10:35.747 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-08-31T22:10:36.083 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-08-31T22:10:36.110 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-08-31T22:10:36.150 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-08-31T22:10:36.529 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-08-31T22:10:36.555 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-08-31T22:10:36.595 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-31T22:10:37.105 INFO:teuthology.orchestra.run.smithi032.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-244-g3091dfc0-1jammy [130 kB] 2024-08-31T22:10:37.107 INFO:teuthology.orchestra.run.smithi032.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-244-g3091dfc0-1jammy [39.0 kB] 2024-08-31T22:10:37.108 INFO:teuthology.orchestra.run.smithi032.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-244-g3091dfc0-1jammy [13.6 MB] 2024-08-31T22:10:37.174 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-08-31T22:10:37.200 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-08-31T22:10:37.240 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-31T22:10:37.546 INFO:teuthology.orchestra.run.smithi032.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-244-g3091dfc0-1jammy [91.4 kB] 2024-08-31T22:10:37.548 INFO:teuthology.orchestra.run.smithi032.stdout:Get:119 https://4.chacra.ceph.com/r/ceph/squid/3091dfc0187f68df2afd4e392dc4c381cfdc1cad/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.1.1-244-g3091dfc0-1jammy [3,140 kB] 2024-08-31T22:10:37.577 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-08-31T22:10:37.603 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-08-31T22:10:37.643 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-31T22:10:37.955 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-08-31T22:10:37.982 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-08-31T22:10:38.013 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-08-31T22:10:38.256 INFO:teuthology.orchestra.run.smithi032.stdout:Fetched 184 MB in 9s (19.4 MB/s) 2024-08-31T22:10:38.300 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package librados2. 2024-08-31T22:10:38.326 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../007-librados2_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:38.366 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking librados2 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:38.540 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-08-31T22:10:38.644 INFO:teuthology.orchestra.run.smithi032.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-08-31T22:10:38.650 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-08-31T22:10:38.738 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-08-31T22:10:38.827 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package librbd1. 2024-08-31T22:10:38.854 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../008-librbd1_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:38.893 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking librbd1 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:39.100 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-08-31T22:10:39.127 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-08-31T22:10:39.167 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-08-31T22:10:39.330 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libcephfs2. 2024-08-31T22:10:39.357 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:39.389 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libcephfs2 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:39.461 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-08-31T22:10:39.488 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-08-31T22:10:39.528 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-08-31T22:10:39.633 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-rados. 2024-08-31T22:10:39.659 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../010-python3-rados_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:39.699 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-rados (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:39.981 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-08-31T22:10:40.011 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-08-31T22:10:40.019 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-08-31T22:10:40.046 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-244-g3091dfc0-1jammy_all.deb ... 2024-08-31T22:10:40.065 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-31T22:10:40.085 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-ceph-argparse (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:40.321 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-cephfs. 2024-08-31T22:10:40.337 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:40.379 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-cephfs (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:40.484 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-08-31T22:10:40.511 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-08-31T22:10:40.551 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-31T22:10:40.862 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-08-31T22:10:40.874 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-ceph-common. 2024-08-31T22:10:40.889 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-08-31T22:10:40.897 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-244-g3091dfc0-1jammy_all.deb ... 2024-08-31T22:10:40.920 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-31T22:10:40.932 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-ceph-common (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:41.219 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-wcwidth. 2024-08-31T22:10:41.245 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-08-31T22:10:41.273 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-08-31T22:10:41.277 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-08-31T22:10:41.300 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-08-31T22:10:41.340 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-08-31T22:10:41.572 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-prettytable. 2024-08-31T22:10:41.598 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-08-31T22:10:41.638 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-08-31T22:10:41.668 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package librados2. 2024-08-31T22:10:41.694 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../007-librados2_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:41.734 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking librados2 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:41.882 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-rbd. 2024-08-31T22:10:41.898 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:41.932 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-rbd (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:42.187 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package librbd1. 2024-08-31T22:10:42.213 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../008-librbd1_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:42.253 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking librbd1 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:42.369 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-08-31T22:10:42.396 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-08-31T22:10:42.435 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-08-31T22:10:42.648 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libcephfs2. 2024-08-31T22:10:42.674 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:42.714 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libcephfs2 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:42.722 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-08-31T22:10:42.748 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-08-31T22:10:42.788 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-08-31T22:10:43.000 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-rados. 2024-08-31T22:10:43.027 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../010-python3-rados_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:43.066 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-rados (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:43.091 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-08-31T22:10:43.107 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-08-31T22:10:43.149 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-08-31T22:10:43.394 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-08-31T22:10:43.421 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-244-g3091dfc0-1jammy_all.deb ... 2024-08-31T22:10:43.461 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-ceph-argparse (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:43.469 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package lua5.1. 2024-08-31T22:10:43.496 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-08-31T22:10:43.536 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-08-31T22:10:43.713 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-cephfs. 2024-08-31T22:10:43.740 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:43.780 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-cephfs (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:43.839 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package lua-any. 2024-08-31T22:10:43.855 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-08-31T22:10:43.897 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking lua-any (27ubuntu1) ... 2024-08-31T22:10:44.057 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-ceph-common. 2024-08-31T22:10:44.073 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-244-g3091dfc0-1jammy_all.deb ... 2024-08-31T22:10:44.107 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-ceph-common (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:44.150 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package zip. 2024-08-31T22:10:44.177 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-08-31T22:10:44.217 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking zip (3.0-12build2) ... 2024-08-31T22:10:44.452 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-wcwidth. 2024-08-31T22:10:44.478 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-08-31T22:10:44.495 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package unzip. 2024-08-31T22:10:44.518 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-08-31T22:10:44.522 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-08-31T22:10:44.553 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-08-31T22:10:44.846 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-prettytable. 2024-08-31T22:10:44.873 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-08-31T22:10:44.912 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-08-31T22:10:44.932 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package luarocks. 2024-08-31T22:10:44.959 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-08-31T22:10:44.999 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-08-31T22:10:45.198 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-rbd. 2024-08-31T22:10:45.225 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:45.265 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-rbd (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:45.359 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package librgw2. 2024-08-31T22:10:45.386 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../025-librgw2_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:45.425 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking librgw2 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:45.601 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-08-31T22:10:45.628 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-08-31T22:10:45.659 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-08-31T22:10:45.920 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-08-31T22:10:45.947 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-08-31T22:10:45.987 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-08-31T22:10:46.228 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-rgw. 2024-08-31T22:10:46.255 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:46.289 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-08-31T22:10:46.294 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-rgw (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:46.305 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-08-31T22:10:46.348 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-08-31T22:10:46.614 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package liboath0:amd64. 2024-08-31T22:10:46.641 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-08-31T22:10:46.681 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-08-31T22:10:46.992 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libradosstriper1. 2024-08-31T22:10:47.045 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:47.045 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package lua5.1. 2024-08-31T22:10:47.045 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-08-31T22:10:47.058 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libradosstriper1 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:47.084 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-08-31T22:10:47.379 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package lua-any. 2024-08-31T22:10:47.403 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-08-31T22:10:47.437 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking lua-any (27ubuntu1) ... 2024-08-31T22:10:47.503 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-common. 2024-08-31T22:10:47.530 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../029-ceph-common_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:47.570 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-common (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:47.665 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package zip. 2024-08-31T22:10:47.691 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-08-31T22:10:47.731 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking zip (3.0-12build2) ... 2024-08-31T22:10:48.051 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package unzip. 2024-08-31T22:10:48.078 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-08-31T22:10:48.117 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-08-31T22:10:48.521 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package luarocks. 2024-08-31T22:10:48.548 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-08-31T22:10:48.587 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-08-31T22:10:48.978 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package librgw2. 2024-08-31T22:10:48.992 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-base. 2024-08-31T22:10:49.005 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../025-librgw2_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:49.018 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../030-ceph-base_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:49.036 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking librgw2 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:49.068 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-base (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:49.589 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-rgw. 2024-08-31T22:10:49.605 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:49.647 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-rgw (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:49.654 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-08-31T22:10:49.681 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-08-31T22:10:49.721 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-08-31T22:10:49.992 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package liboath0:amd64. 2024-08-31T22:10:50.021 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-08-31T22:10:50.023 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-cheroot. 2024-08-31T22:10:50.051 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-08-31T22:10:50.058 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-08-31T22:10:50.090 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-08-31T22:10:50.378 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libradosstriper1. 2024-08-31T22:10:50.404 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:50.409 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-08-31T22:10:50.437 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-08-31T22:10:50.444 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libradosstriper1 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:50.476 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-08-31T22:10:50.754 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jaraco.text. 2024-08-31T22:10:50.770 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-08-31T22:10:50.797 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-common. 2024-08-31T22:10:50.804 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-08-31T22:10:50.824 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../029-ceph-common_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:50.855 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-common (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:51.107 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-08-31T22:10:51.134 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-08-31T22:10:51.173 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-08-31T22:10:51.459 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-tempora. 2024-08-31T22:10:51.486 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-08-31T22:10:51.542 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-08-31T22:10:51.829 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-portend. 2024-08-31T22:10:51.856 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-08-31T22:10:51.895 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-08-31T22:10:52.173 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-08-31T22:10:52.200 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-08-31T22:10:52.240 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-08-31T22:10:52.344 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-base. 2024-08-31T22:10:52.371 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../030-ceph-base_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:10:52.428 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-base (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:10:52.543 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-cherrypy3. 2024-08-31T22:10:52.569 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-08-31T22:10:52.609 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-08-31T22:10:52.929 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-natsort. 2024-08-31T22:10:52.945 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-08-31T22:10:52.979 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-08-31T22:10:53.081 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-08-31T22:10:53.108 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-08-31T22:10:53.156 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-08-31T22:10:53.291 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-logutils. 2024-08-31T22:10:53.318 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-08-31T22:10:53.357 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-08-31T22:10:53.476 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-cheroot. 2024-08-31T22:10:53.502 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-08-31T22:10:53.542 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-08-31T22:10:53.643 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-markupsafe. 2024-08-31T22:10:53.660 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-08-31T22:10:53.701 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-08-31T22:10:53.879 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-08-31T22:10:53.905 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-08-31T22:10:53.945 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-08-31T22:10:53.962 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-mako. 2024-08-31T22:10:53.989 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-08-31T22:10:54.029 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-08-31T22:10:54.281 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-jaraco.text. 2024-08-31T22:10:54.308 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-08-31T22:10:54.316 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-simplegeneric. 2024-08-31T22:10:54.343 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-08-31T22:10:54.356 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-08-31T22:10:54.374 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-08-31T22:10:54.652 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-singledispatch. 2024-08-31T22:10:54.678 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-08-31T22:10:54.718 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-08-31T22:10:54.734 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-08-31T22:10:54.761 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-08-31T22:10:54.809 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-08-31T22:10:55.021 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-webob. 2024-08-31T22:10:55.047 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-08-31T22:10:55.087 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-08-31T22:10:55.145 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-tempora. 2024-08-31T22:10:55.172 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-08-31T22:10:55.211 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-08-31T22:10:55.432 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-waitress. 2024-08-31T22:10:55.459 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-08-31T22:10:55.502 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-08-31T22:10:55.514 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-portend. 2024-08-31T22:10:55.541 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-08-31T22:10:55.581 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-08-31T22:10:55.835 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-tempita. 2024-08-31T22:10:55.864 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-08-31T22:10:55.901 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-08-31T22:10:55.959 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-08-31T22:10:55.985 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-08-31T22:10:56.042 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-08-31T22:10:56.338 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-paste. 2024-08-31T22:10:56.364 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-08-31T22:10:56.404 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-08-31T22:10:56.420 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-cherrypy3. 2024-08-31T22:10:56.447 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-08-31T22:10:56.494 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-08-31T22:10:56.741 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-08-31T22:10:56.768 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-08-31T22:10:56.806 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-natsort. 2024-08-31T22:10:56.807 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-08-31T22:10:56.823 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-08-31T22:10:56.856 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-08-31T22:10:57.077 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pastedeploy. 2024-08-31T22:10:57.104 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-08-31T22:10:57.143 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-08-31T22:10:57.159 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-logutils. 2024-08-31T22:10:57.187 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-08-31T22:10:57.226 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-08-31T22:10:57.446 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-webtest. 2024-08-31T22:10:57.473 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-08-31T22:10:57.512 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-08-31T22:10:57.537 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-markupsafe. 2024-08-31T22:10:57.553 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-08-31T22:10:57.595 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-08-31T22:10:57.807 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pecan. 2024-08-31T22:10:57.834 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-08-31T22:10:57.873 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-08-31T22:10:57.898 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-mako. 2024-08-31T22:10:57.925 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-08-31T22:10:57.964 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-08-31T22:10:58.237 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-certifi. 2024-08-31T22:10:58.264 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-08-31T22:10:58.267 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-simplegeneric. 2024-08-31T22:10:58.283 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-08-31T22:10:58.303 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-08-31T22:10:58.317 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-08-31T22:10:58.579 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-singledispatch. 2024-08-31T22:10:58.605 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-08-31T22:10:58.631 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-idna. 2024-08-31T22:10:58.645 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-08-31T22:10:58.658 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-08-31T22:10:58.706 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-08-31T22:10:58.947 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-webob. 2024-08-31T22:10:58.974 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-08-31T22:10:58.975 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-urllib3. 2024-08-31T22:10:59.002 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-08-31T22:10:59.014 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-08-31T22:10:59.042 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-08-31T22:10:59.317 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-waitress. 2024-08-31T22:10:59.328 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-requests. 2024-08-31T22:10:59.333 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-08-31T22:10:59.355 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-08-31T22:10:59.378 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-08-31T22:10:59.403 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-08-31T22:10:59.686 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-tempita. 2024-08-31T22:10:59.698 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-werkzeug. 2024-08-31T22:10:59.714 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-08-31T22:10:59.725 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-08-31T22:10:59.753 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-08-31T22:10:59.773 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-08-31T22:11:00.042 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-08-31T22:11:00.056 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-paste. 2024-08-31T22:11:00.058 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-244-g3091dfc0-1jammy_all.deb ... 2024-08-31T22:11:00.083 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-08-31T22:11:00.092 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-modules-core (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:00.122 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-08-31T22:11:00.472 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-08-31T22:11:00.483 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-08-31T22:11:00.499 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:00.500 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-08-31T22:11:00.538 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:00.542 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-08-31T22:11:00.853 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pastedeploy. 2024-08-31T22:11:00.858 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-bcrypt. 2024-08-31T22:11:00.880 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-08-31T22:11:00.885 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-08-31T22:11:00.919 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-08-31T22:11:00.924 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-08-31T22:11:01.219 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-openssl. 2024-08-31T22:11:01.222 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-webtest. 2024-08-31T22:11:01.246 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-08-31T22:11:01.250 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-08-31T22:11:01.669 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-08-31T22:11:01.706 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-08-31T22:11:02.330 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr. 2024-08-31T22:11:02.355 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:02.388 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:02.759 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pecan. 2024-08-31T22:11:02.784 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mon. 2024-08-31T22:11:02.785 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-08-31T22:11:02.811 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:02.825 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-08-31T22:11:02.850 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mon (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:03.180 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-certifi. 2024-08-31T22:11:03.207 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-08-31T22:11:03.246 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-08-31T22:11:03.456 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-osd. 2024-08-31T22:11:03.481 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:03.520 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-osd (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:03.582 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-idna. 2024-08-31T22:11:03.609 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-08-31T22:11:03.649 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-08-31T22:11:03.927 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-urllib3. 2024-08-31T22:11:03.943 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-08-31T22:11:03.976 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-08-31T22:11:04.246 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-requests. 2024-08-31T22:11:04.276 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-08-31T22:11:04.304 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-08-31T22:11:04.591 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-werkzeug. 2024-08-31T22:11:04.618 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-08-31T22:11:04.631 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph. 2024-08-31T22:11:04.648 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../066-ceph_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:04.665 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-08-31T22:11:04.681 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:04.952 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-08-31T22:11:04.968 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-fuse. 2024-08-31T22:11:04.968 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-244-g3091dfc0-1jammy_all.deb ... 2024-08-31T22:11:04.995 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:05.001 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mgr-modules-core (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:05.034 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-fuse (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:05.396 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mds. 2024-08-31T22:11:05.398 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-08-31T22:11:05.422 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:05.425 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:05.462 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mds (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:05.464 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:05.750 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-bcrypt. 2024-08-31T22:11:05.766 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-08-31T22:11:05.800 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-08-31T22:11:05.856 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package cephadm. 2024-08-31T22:11:05.884 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../069-cephadm_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:05.923 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking cephadm (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:06.070 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-openssl. 2024-08-31T22:11:06.097 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-08-31T22:11:06.170 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-08-31T22:11:06.192 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-asyncssh. 2024-08-31T22:11:06.208 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-08-31T22:11:06.242 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-08-31T22:11:06.514 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mgr. 2024-08-31T22:11:06.537 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python-babel-localedata. 2024-08-31T22:11:06.541 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:06.564 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-08-31T22:11:06.580 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mgr (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:06.603 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-08-31T22:11:06.984 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mon. 2024-08-31T22:11:07.011 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:07.050 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mon (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:07.629 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-osd. 2024-08-31T22:11:07.656 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:07.695 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-osd (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:07.792 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-babel. 2024-08-31T22:11:07.819 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-08-31T22:11:07.858 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-08-31T22:11:08.203 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jinja2. 2024-08-31T22:11:08.230 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-08-31T22:11:08.269 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-08-31T22:11:08.597 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-08-31T22:11:08.624 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-244-g3091dfc0-1jammy_all.deb ... 2024-08-31T22:11:08.663 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-cephadm (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:08.773 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph. 2024-08-31T22:11:08.789 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../066-ceph_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:08.831 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:09.017 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-repoze.lru. 2024-08-31T22:11:09.047 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-08-31T22:11:09.083 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-08-31T22:11:09.151 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-fuse. 2024-08-31T22:11:09.178 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:09.217 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-fuse (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:09.336 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-routes. 2024-08-31T22:11:09.353 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-08-31T22:11:09.394 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-08-31T22:11:09.529 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mds. 2024-08-31T22:11:09.556 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:09.587 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mds (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:09.706 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-08-31T22:11:09.733 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-244-g3091dfc0-1jammy_all.deb ... 2024-08-31T22:11:09.772 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-dashboard (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:09.990 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package cephadm. 2024-08-31T22:11:10.017 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../069-cephadm_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:10.056 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking cephadm (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:10.375 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-asyncssh. 2024-08-31T22:11:10.403 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-08-31T22:11:10.442 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-08-31T22:11:10.795 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python-babel-localedata. 2024-08-31T22:11:10.811 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-08-31T22:11:10.853 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-08-31T22:11:11.086 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-08-31T22:11:11.102 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-08-31T22:11:11.144 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-08-31T22:11:11.631 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-joblib. 2024-08-31T22:11:11.658 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-08-31T22:11:11.697 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-08-31T22:11:11.816 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-babel. 2024-08-31T22:11:11.844 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-08-31T22:11:11.882 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-08-31T22:11:12.017 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-08-31T22:11:12.033 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-08-31T22:11:12.077 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-08-31T22:11:12.210 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-jinja2. 2024-08-31T22:11:12.237 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-08-31T22:11:12.318 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-08-31T22:11:12.403 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-sklearn. 2024-08-31T22:11:12.431 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-08-31T22:11:12.470 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-08-31T22:11:12.671 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-08-31T22:11:12.699 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-244-g3091dfc0-1jammy_all.deb ... 2024-08-31T22:11:12.738 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mgr-cephadm (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:13.049 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-repoze.lru. 2024-08-31T22:11:13.077 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-08-31T22:11:13.116 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-08-31T22:11:13.178 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-08-31T22:11:13.206 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-244-g3091dfc0-1jammy_all.deb ... 2024-08-31T22:11:13.330 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:13.418 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-routes. 2024-08-31T22:11:13.446 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-08-31T22:11:13.493 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-08-31T22:11:13.830 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-08-31T22:11:13.857 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-244-g3091dfc0-1jammy_all.deb ... 2024-08-31T22:11:13.896 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mgr-dashboard (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:14.200 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-cachetools. 2024-08-31T22:11:14.227 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-08-31T22:11:14.266 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-08-31T22:11:14.569 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pyasn1. 2024-08-31T22:11:14.597 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-08-31T22:11:14.635 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-08-31T22:11:14.972 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-08-31T22:11:15.000 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-08-31T22:11:15.038 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-08-31T22:11:15.192 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-08-31T22:11:15.209 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-08-31T22:11:15.251 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-08-31T22:11:15.350 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-rsa. 2024-08-31T22:11:15.378 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-08-31T22:11:15.416 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-rsa (4.8-1) ... 2024-08-31T22:11:15.723 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-joblib. 2024-08-31T22:11:15.737 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-google-auth. 2024-08-31T22:11:15.739 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-08-31T22:11:15.765 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-08-31T22:11:15.773 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-08-31T22:11:15.803 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-08-31T22:11:16.106 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-08-31T22:11:16.123 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-08-31T22:11:16.143 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-08-31T22:11:16.164 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-08-31T22:11:16.170 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-08-31T22:11:16.209 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-08-31T22:11:16.467 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-websocket. 2024-08-31T22:11:16.495 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-08-31T22:11:16.520 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-sklearn. 2024-08-31T22:11:16.534 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-08-31T22:11:16.548 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-08-31T22:11:16.587 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-08-31T22:11:16.862 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-kubernetes. 2024-08-31T22:11:16.890 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-08-31T22:11:16.928 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-08-31T22:11:17.337 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-08-31T22:11:17.366 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-244-g3091dfc0-1jammy_all.deb ... 2024-08-31T22:11:17.534 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-08-31T22:11:17.551 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-244-g3091dfc0-1jammy_all.deb ... 2024-08-31T22:11:17.592 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:17.706 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:18.112 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libonig5:amd64. 2024-08-31T22:11:18.140 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-08-31T22:11:18.178 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-08-31T22:11:18.514 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libjq1:amd64. 2024-08-31T22:11:18.542 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-08-31T22:11:18.581 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-08-31T22:11:18.659 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-cachetools. 2024-08-31T22:11:18.687 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-08-31T22:11:18.726 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-08-31T22:11:18.842 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package jq. 2024-08-31T22:11:18.867 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-08-31T22:11:18.908 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-08-31T22:11:18.987 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pyasn1. 2024-08-31T22:11:19.003 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-08-31T22:11:19.037 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-08-31T22:11:19.111 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package socat. 2024-08-31T22:11:19.127 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-08-31T22:11:19.169 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-08-31T22:11:19.382 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-08-31T22:11:19.410 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-08-31T22:11:19.441 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-08-31T22:11:19.447 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package xmlstarlet. 2024-08-31T22:11:19.463 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-08-31T22:11:19.505 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-08-31T22:11:19.710 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-rsa. 2024-08-31T22:11:19.737 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-08-31T22:11:19.768 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-rsa (4.8-1) ... 2024-08-31T22:11:19.859 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-test. 2024-08-31T22:11:19.876 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../097-ceph-test_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:19.918 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-test (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:20.097 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-google-auth. 2024-08-31T22:11:20.125 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-08-31T22:11:20.163 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-08-31T22:11:20.483 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-08-31T22:11:20.511 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-08-31T22:11:20.558 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-08-31T22:11:20.936 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-websocket. 2024-08-31T22:11:20.963 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-08-31T22:11:21.011 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-08-31T22:11:21.322 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-kubernetes. 2024-08-31T22:11:21.350 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-08-31T22:11:21.380 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-08-31T22:11:22.078 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-08-31T22:11:22.106 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-244-g3091dfc0-1jammy_all.deb ... 2024-08-31T22:11:22.152 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:22.238 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-volume. 2024-08-31T22:11:22.266 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-244-g3091dfc0-1jammy_all.deb ... 2024-08-31T22:11:22.305 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-volume (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:22.572 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libonig5:amd64. 2024-08-31T22:11:22.600 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-08-31T22:11:22.647 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-08-31T22:11:22.734 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libcephfs-dev. 2024-08-31T22:11:22.762 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:22.800 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libcephfs-dev (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:23.103 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package lua-socket:amd64. 2024-08-31T22:11:23.125 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libjq1:amd64. 2024-08-31T22:11:23.131 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-08-31T22:11:23.141 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-08-31T22:11:23.169 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-08-31T22:11:23.174 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-08-31T22:11:23.436 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package jq. 2024-08-31T22:11:23.464 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-08-31T22:11:23.502 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-08-31T22:11:23.524 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package lua-sec:amd64. 2024-08-31T22:11:23.553 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-08-31T22:11:23.582 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-08-31T22:11:23.772 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package socat. 2024-08-31T22:11:23.800 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-08-31T22:11:23.838 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-08-31T22:11:23.953 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package nvme-cli. 2024-08-31T22:11:23.981 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-08-31T22:11:24.019 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-08-31T22:11:24.150 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package xmlstarlet. 2024-08-31T22:11:24.178 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-08-31T22:11:24.216 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-08-31T22:11:24.425 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package pkg-config. 2024-08-31T22:11:24.453 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-08-31T22:11:24.491 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-08-31T22:11:24.604 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-test. 2024-08-31T22:11:24.632 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../097-ceph-test_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:24.670 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-test (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:24.777 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-08-31T22:11:24.806 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-08-31T22:11:24.844 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-08-31T22:11:25.263 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-iniconfig. 2024-08-31T22:11:25.291 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-08-31T22:11:25.329 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-08-31T22:11:25.626 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pastescript. 2024-08-31T22:11:25.654 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-08-31T22:11:25.692 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-08-31T22:11:26.013 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pluggy. 2024-08-31T22:11:26.041 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-08-31T22:11:26.079 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-08-31T22:11:26.391 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-psutil. 2024-08-31T22:11:26.419 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-08-31T22:11:26.457 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-08-31T22:11:26.802 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-py. 2024-08-31T22:11:26.830 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-08-31T22:11:26.877 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-py (1.10.0-1) ... 2024-08-31T22:11:27.083 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-volume. 2024-08-31T22:11:27.111 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-244-g3091dfc0-1jammy_all.deb ... 2024-08-31T22:11:27.149 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-volume (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:27.197 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pygments. 2024-08-31T22:11:27.225 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-08-31T22:11:27.263 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-08-31T22:11:27.495 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libcephfs-dev. 2024-08-31T22:11:27.511 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:27.553 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libcephfs-dev (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:27.718 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pyinotify. 2024-08-31T22:11:27.747 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-08-31T22:11:27.784 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-08-31T22:11:28.054 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-toml. 2024-08-31T22:11:28.056 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package lua-socket:amd64. 2024-08-31T22:11:28.082 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-08-31T22:11:28.084 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-08-31T22:11:28.112 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-08-31T22:11:28.131 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-08-31T22:11:28.415 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pytest. 2024-08-31T22:11:28.419 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package lua-sec:amd64. 2024-08-31T22:11:28.436 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-08-31T22:11:28.443 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-08-31T22:11:28.469 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-08-31T22:11:28.481 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-08-31T22:11:28.819 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-simplejson. 2024-08-31T22:11:28.847 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-08-31T22:11:28.856 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package nvme-cli. 2024-08-31T22:11:28.884 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-08-31T22:11:28.884 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-08-31T22:11:28.922 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-08-31T22:11:29.104 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-08-31T22:11:29.121 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-08-31T22:11:29.154 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-08-31T22:11:29.394 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package pkg-config. 2024-08-31T22:11:29.422 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-08-31T22:11:29.469 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-08-31T22:11:29.781 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-08-31T22:11:29.809 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-08-31T22:11:29.833 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package radosgw. 2024-08-31T22:11:29.855 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-08-31T22:11:29.862 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../116-radosgw_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:29.899 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking radosgw (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:30.250 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-iniconfig. 2024-08-31T22:11:30.278 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-08-31T22:11:30.308 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-08-31T22:11:30.586 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pastescript. 2024-08-31T22:11:30.614 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-08-31T22:11:30.652 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-08-31T22:11:30.728 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package rbd-fuse. 2024-08-31T22:11:30.745 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:30.786 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking rbd-fuse (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:30.981 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pluggy. 2024-08-31T22:11:31.009 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-08-31T22:11:31.047 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-08-31T22:11:31.147 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package rbd-mirror. 2024-08-31T22:11:31.176 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../118-rbd-mirror_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:31.214 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking rbd-mirror (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:31.300 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-psutil. 2024-08-31T22:11:31.328 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-08-31T22:11:31.366 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-08-31T22:11:31.627 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-08-31T22:11:31.686 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-py. 2024-08-31T22:11:31.714 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-08-31T22:11:31.752 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-py (1.10.0-1) ... 2024-08-31T22:11:31.903 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-08-31T22:11:32.029 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-08-31T22:11:32.031 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pygments. 2024-08-31T22:11:32.060 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-08-31T22:11:32.089 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-08-31T22:11:32.294 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-08-31T22:11:32.602 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pyinotify. 2024-08-31T22:11:32.614 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-08-31T22:11:32.630 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-08-31T22:11:32.668 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-08-31T22:11:32.979 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-toml. 2024-08-31T22:11:33.007 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-08-31T22:11:33.045 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-08-31T22:11:33.065 INFO:teuthology.orchestra.run.smithi007.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-08-31T22:11:33.172 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up cephadm (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:33.307 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pytest. 2024-08-31T22:11:33.335 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-08-31T22:11:33.365 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-08-31T22:11:33.702 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-simplejson. 2024-08-31T22:11:33.730 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-08-31T22:11:33.768 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-08-31T22:11:33.939 INFO:teuthology.orchestra.run.smithi007.stdout:Adding system user cephadm....done 2024-08-31T22:11:34.029 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-08-31T22:11:34.057 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-08-31T22:11:34.095 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-08-31T22:11:34.105 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-08-31T22:11:34.590 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-08-31T22:11:34.792 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package radosgw. 2024-08-31T22:11:34.820 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../116-radosgw_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:34.858 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking radosgw (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:35.025 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-08-31T22:11:35.160 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-08-31T22:11:35.419 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-08-31T22:11:35.687 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-08-31T22:11:35.694 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package rbd-fuse. 2024-08-31T22:11:35.710 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:35.752 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking rbd-fuse (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:35.813 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-py (1.10.0-1) ... 2024-08-31T22:11:36.105 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package rbd-mirror. 2024-08-31T22:11:36.122 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../118-rbd-mirror_19.1.1-244-g3091dfc0-1jammy_amd64.deb ... 2024-08-31T22:11:36.123 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-08-31T22:11:36.172 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking rbd-mirror (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:36.509 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-08-31T22:11:36.652 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-08-31T22:11:36.768 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-08-31T22:11:36.894 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-08-31T22:11:37.020 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-08-31T22:11:37.078 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-08-31T22:11:37.269 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-08-31T22:11:37.350 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-08-31T22:11:37.571 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-08-31T22:11:37.705 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-08-31T22:11:37.965 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-08-31T22:11:38.000 INFO:teuthology.orchestra.run.smithi032.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-08-31T22:11:38.072 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up cephadm (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:38.224 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-ceph-argparse (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:38.467 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-08-31T22:11:38.496 INFO:teuthology.orchestra.run.smithi032.stdout:Adding system user cephadm....done 2024-08-31T22:11:38.593 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-08-31T22:11:38.661 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-08-31T22:11:38.920 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-08-31T22:11:38.937 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-08-31T22:11:39.037 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-08-31T22:11:39.197 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-08-31T22:11:39.280 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-08-31T22:11:39.323 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-08-31T22:11:39.582 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-08-31T22:11:39.714 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-08-31T22:11:39.735 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-08-31T22:11:39.791 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-08-31T22:11:39.833 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-08-31T22:11:39.926 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-08-31T22:11:39.959 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-py (1.10.0-1) ... 2024-08-31T22:11:40.269 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-08-31T22:11:40.294 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-08-31T22:11:40.420 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-08-31T22:11:40.629 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-08-31T22:11:40.704 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-08-31T22:11:40.881 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-08-31T22:11:40.972 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-08-31T22:11:41.174 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-08-31T22:11:41.282 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up zip (3.0-12build2) ... 2024-08-31T22:11:41.383 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-08-31T22:11:41.409 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-08-31T22:11:41.660 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-08-31T22:11:41.927 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-08-31T22:11:42.093 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-08-31T22:11:42.178 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-ceph-argparse (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:42.361 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-08-31T22:11:42.446 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-08-31T22:11:42.487 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-08-31T22:11:42.589 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-08-31T22:11:42.613 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-08-31T22:11:42.715 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-08-31T22:11:42.833 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-08-31T22:11:42.914 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-08-31T22:11:43.075 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-08-31T22:11:43.307 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-08-31T22:11:43.436 INFO:teuthology.orchestra.run.smithi032.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-08-31T22:11:43.474 INFO:teuthology.orchestra.run.smithi032.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-08-31T22:11:43.537 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-08-31T22:11:43.550 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-08-31T22:11:43.638 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-08-31T22:11:43.977 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-08-31T22:11:43.997 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-08-31T22:11:44.123 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-08-31T22:11:44.278 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-08-31T22:11:44.383 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-08-31T22:11:44.629 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-08-31T22:11:44.642 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-08-31T22:11:44.880 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-08-31T22:11:44.927 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up zip (3.0-12build2) ... 2024-08-31T22:11:44.998 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-ceph-common (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:45.053 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-08-31T22:11:45.316 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-08-31T22:11:45.659 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-08-31T22:11:45.754 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-08-31T22:11:46.022 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-08-31T22:11:46.148 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-08-31T22:11:46.274 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-08-31T22:11:46.575 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-08-31T22:11:46.915 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-08-31T22:11:46.952 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-08-31T22:11:47.175 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-31T22:11:47.202 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-08-31T22:11:47.318 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-toml (0.10.2-1) ... 2024-08-31T22:11:47.594 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-08-31T22:11:47.612 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-08-31T22:11:47.720 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-08-31T22:11:47.846 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-08-31T22:11:47.922 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-08-31T22:11:48.164 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-08-31T22:11:48.265 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-08-31T22:11:48.432 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-08-31T22:11:48.683 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-31T22:11:48.725 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-08-31T22:11:48.809 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-08-31T22:11:48.825 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-ceph-common (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:49.118 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-08-31T22:11:49.169 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-08-31T22:11:49.436 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-08-31T22:11:49.437 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-08-31T22:11:49.696 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-08-31T22:11:49.956 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-08-31T22:11:50.207 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-08-31T22:11:50.458 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-08-31T22:11:50.584 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-08-31T22:11:50.650 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-08-31T22:11:50.869 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-08-31T22:11:50.918 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-31T22:11:50.995 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-08-31T22:11:51.053 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-toml (0.10.2-1) ... 2024-08-31T22:11:51.296 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-08-31T22:11:51.320 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-08-31T22:11:51.446 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-08-31T22:11:51.573 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-08-31T22:11:51.638 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-08-31T22:11:51.882 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-08-31T22:11:51.898 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-08-31T22:11:52.024 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-08-31T22:11:52.133 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-08-31T22:11:52.150 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-08-31T22:11:52.247 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-08-31T22:11:52.368 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-31T22:11:52.485 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-08-31T22:11:52.493 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-08-31T22:11:52.611 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-08-31T22:11:52.820 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-08-31T22:11:53.004 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-08-31T22:11:53.063 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-08-31T22:11:53.264 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up lua-any (27ubuntu1) ... 2024-08-31T22:11:53.322 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-08-31T22:11:53.381 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-portend (3.0.0-1) ... 2024-08-31T22:11:53.598 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-08-31T22:11:53.632 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-31T22:11:53.767 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-08-31T22:11:53.857 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-08-31T22:11:54.093 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-08-31T22:11:54.125 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-08-31T22:11:54.268 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-08-31T22:11:54.478 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-08-31T22:11:54.560 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-08-31T22:11:54.687 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-08-31T22:11:54.754 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-08-31T22:11:54.880 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-rsa (4.8-1) ... 2024-08-31T22:11:54.996 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-08-31T22:11:55.173 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-08-31T22:11:55.289 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-08-31T22:11:55.441 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-08-31T22:11:55.548 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-08-31T22:11:55.674 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-08-31T22:11:55.777 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-08-31T22:11:55.800 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-08-31T22:11:55.890 INFO:teuthology.orchestra.run.smithi032.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-08-31T22:11:56.179 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-08-31T22:11:56.185 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-08-31T22:11:56.311 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-08-31T22:11:56.472 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-08-31T22:11:56.704 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-08-31T22:11:56.833 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-08-31T22:11:56.956 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up lua-any (27ubuntu1) ... 2024-08-31T22:11:56.960 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up librados2 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:57.081 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-portend (3.0.0-1) ... 2024-08-31T22:11:57.086 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libsqlite3-mod-ceph (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:57.212 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-08-31T22:11:57.357 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-31T22:11:57.483 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-08-31T22:11:57.573 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libcephfs2 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:57.674 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libradosstriper1 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:57.784 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-08-31T22:11:57.809 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-08-31T22:11:58.069 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up librbd1 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:58.127 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-08-31T22:11:58.195 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-modules-core (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:58.296 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-fuse (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:58.403 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-08-31T22:11:58.506 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-08-31T22:11:58.506 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-08-31T22:11:58.529 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-rsa (4.8-1) ... 2024-08-31T22:11:58.822 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-08-31T22:11:59.014 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libcephfs-dev (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:59.132 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-08-31T22:11:59.156 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-rados (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:59.282 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up librgw2 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:11:59.417 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-08-31T22:11:59.477 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-08-31T22:11:59.870 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-08-31T22:12:00.147 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-08-31T22:12:00.524 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-08-31T22:12:00.625 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up librados2 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:00.687 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-rbd (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:00.734 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libsqlite3-mod-ceph (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:00.813 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up rbd-fuse (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:00.860 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-08-31T22:12:00.931 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-rgw (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:01.032 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-cephfs (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:01.133 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-common (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:01.197 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libcephfs2 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:01.331 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libradosstriper1 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:01.449 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-08-31T22:12:01.710 INFO:teuthology.orchestra.run.smithi007.stdout:Adding group ceph....done 2024-08-31T22:12:01.717 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up librbd1 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:01.843 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mgr-modules-core (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:01.969 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-fuse (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:02.094 INFO:teuthology.orchestra.run.smithi007.stdout:Adding system user ceph....done 2024-08-31T22:12:02.180 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-08-31T22:12:02.180 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-08-31T22:12:02.264 INFO:teuthology.orchestra.run.smithi007.stdout:Setting system user ceph properties....done 2024-08-31T22:12:02.283 INFO:teuthology.orchestra.run.smithi007.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-08-31T22:12:02.442 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-08-31T22:12:02.653 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libcephfs-dev (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:02.800 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-08-31T22:12:02.805 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-rados (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:02.933 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up librgw2 (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:03.034 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-08-31T22:12:03.310 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-test (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:03.452 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up radosgw (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:03.916 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-08-31T22:12:03.916 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-08-31T22:12:04.246 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-rbd (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:04.372 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up rbd-fuse (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:04.379 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-base (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:04.498 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-rgw (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:04.624 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-cephfs (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:04.734 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-common (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:04.743 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-08-31T22:12:05.215 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mds (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:05.235 INFO:teuthology.orchestra.run.smithi032.stdout:Adding group ceph....done 2024-08-31T22:12:05.436 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-08-31T22:12:05.436 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-08-31T22:12:05.646 INFO:teuthology.orchestra.run.smithi032.stdout:Adding system user ceph....done 2024-08-31T22:12:05.806 INFO:teuthology.orchestra.run.smithi032.stdout:Setting system user ceph properties....done 2024-08-31T22:12:05.823 INFO:teuthology.orchestra.run.smithi032.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-08-31T22:12:05.966 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-08-31T22:12:05.967 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up rbd-mirror (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:06.180 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-08-31T22:12:06.180 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-08-31T22:12:06.312 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-08-31T22:12:06.703 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:06.793 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-test (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:06.895 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up radosgw (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:06.976 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-08-31T22:12:06.976 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-08-31T22:12:07.349 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-08-31T22:12:07.349 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-08-31T22:12:07.471 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-osd (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:07.768 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-08-31T22:12:07.768 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-08-31T22:12:07.805 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-base (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:08.176 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-08-31T22:12:08.265 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-k8sevents (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:08.391 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:08.551 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mon (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:08.657 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mds (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:08.767 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-08-31T22:12:08.767 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-08-31T22:12:08.908 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-08-31T22:12:08.908 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-08-31T22:12:09.244 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-cephadm (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:09.376 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up rbd-mirror (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:09.395 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:09.521 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-dashboard (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:09.598 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-08-31T22:12:09.598 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-08-31T22:12:10.015 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-volume (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:10.093 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mgr (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:10.322 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-08-31T22:12:10.322 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-08-31T22:12:10.385 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-08-31T22:12:10.486 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-08-31T22:12:10.632 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-08-31T22:12:10.795 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-osd (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:11.059 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-08-31T22:12:11.059 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-08-31T22:12:11.421 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-08-31T22:12:11.547 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mgr-k8sevents (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:11.672 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:11.840 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mon (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:12.071 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-08-31T22:12:12.071 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-08-31T22:12:12.092 INFO:teuthology.orchestra.run.smithi007.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-08-31T22:12:12.099 DEBUG:teuthology.parallel:result is None 2024-08-31T22:12:12.533 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mgr-cephadm (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:12.668 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:12.794 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mgr-dashboard (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:12.932 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-volume (19.1.1-244-g3091dfc0-1jammy) ... 2024-08-31T22:12:13.281 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-08-31T22:12:13.390 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-08-31T22:12:13.535 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-08-31T22:12:15.005 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-08-31T22:12:15.680 INFO:teuthology.orchestra.run.smithi032.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-08-31T22:12:15.685 DEBUG:teuthology.parallel:result is None 2024-08-31T22:12:15.686 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3091dfc0187f68df2afd4e392dc4c381cfdc1cad 2024-08-31T22:12:15.828 DEBUG:teuthology.orchestra.run.smithi007:> dpkg-query -W -f '${Version}' ceph 2024-08-31T22:12:15.848 INFO:teuthology.orchestra.run.smithi007.stdout:19.1.1-244-g3091dfc0-1jammy 2024-08-31T22:12:15.848 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-244-g3091dfc0-1jammy 2024-08-31T22:12:15.849 INFO:teuthology.task.install:The correct ceph version 19.1.1-244-g3091dfc0-1jammy is installed. 2024-08-31T22:12:15.850 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3091dfc0187f68df2afd4e392dc4c381cfdc1cad 2024-08-31T22:12:15.990 DEBUG:teuthology.orchestra.run.smithi032:> dpkg-query -W -f '${Version}' ceph 2024-08-31T22:12:16.010 INFO:teuthology.orchestra.run.smithi032.stdout:19.1.1-244-g3091dfc0-1jammy 2024-08-31T22:12:16.011 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-244-g3091dfc0-1jammy 2024-08-31T22:12:16.011 INFO:teuthology.task.install:The correct ceph version 19.1.1-244-g3091dfc0-1jammy is installed. 2024-08-31T22:12:16.012 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-31T22:12:16.013 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:16.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-31T22:12:16.028 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:16.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-31T22:12:16.066 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-31T22:12:16.066 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:16.067 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/daemon-helper 2024-08-31T22:12:16.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-31T22:12:16.136 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:16.136 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/daemon-helper 2024-08-31T22:12:16.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-31T22:12:16.205 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-31T22:12:16.205 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:16.205 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-31T22:12:16.218 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-31T22:12:16.272 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:16.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-31T22:12:16.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-31T22:12:16.341 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-31T22:12:16.341 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:16.341 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/stdin-killer 2024-08-31T22:12:16.354 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-31T22:12:16.408 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:16.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/stdin-killer 2024-08-31T22:12:16.421 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-31T22:12:16.476 INFO:teuthology.run_tasks:Running task ceph... 2024-08-31T22:12:16.552 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-08-31T22:12:16.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /var/log/ceph 2024-08-31T22:12:16.554 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /var/log/ceph 2024-08-31T22:12:16.568 INFO:tasks.ceph:Disabling ceph logrotate... 2024-08-31T22:12:16.568 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-31T22:12:16.610 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-31T22:12:16.623 INFO:tasks.ceph:Creating extra log directories... 2024-08-31T22:12:16.623 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-31T22:12:16.666 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-31T22:12:16.693 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-08-31T22:12:16.693 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-08-31T22:12:16.693 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-08-31_21:16:02-rbd-squid-distro-default-smithi/7883608', 'branch': 'squid', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7883608', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-08-31_21:16:02-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '3091dfc0187f68df2afd4e392dc4c381cfdc1cad'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3091dfc0187f68df2afd4e392dc4c381cfdc1cad'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': '3091dfc0187f68df2afd4e392dc4c381cfdc1cad'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 552, 'sha1': '3091dfc0187f68df2afd4e392dc4c381cfdc1cad', 'sleep_before_teardown': 0, 'subset': '40/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_3091dfc0187f68df2afd4e392dc4c381cfdc1cad/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '3091dfc0187f68df2afd4e392dc4c381cfdc1cad', 'targets': {'smithi007.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPXUeOWcSMUCUypX4YCMcgYYBi9GKauqGGUmH0fi5vzIeU6SJbiht+tepppB/MtJQna3sVZ2fKQUW0qywuwh218=', 'smithi032.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC9ImDX2yUmt3UPP3yhZ5c5Z5ePr116mvTICgiHMVQJB4QGQf95OWkNXcWQYcyswUbvK1EMz1+9d/gLndRc9F3M='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '3091dfc0187f68df2afd4e392dc4c381cfdc1cad'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '3091dfc0187f68df2afd4e392dc4c381cfdc1cad'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_MIN_COMPAT_CLIENT': 'octopus', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-08-31_21:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-08-31T22:12:16.693 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-08-31T22:12:16.722 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-08-31T22:12:16.730 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/run/ceph 2024-08-31T22:12:16.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m0777 -- /var/run/ceph 2024-08-31T22:12:16.786 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:16.786 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2024-08-31T22:12:16.829 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-31T22:12:16.829 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2024-08-31T22:12:16.877 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-31T22:12:16.877 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-31T22:12:16.877 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 863 Links: 1 2024-08-31T22:12:16.877 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-31T22:12:16.877 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-08-31 22:06:35.069519861 +0000 2024-08-31T22:12:16.878 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-08-31 22:06:34.813522134 +0000 2024-08-31T22:12:16.878 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-08-31 22:06:34.813522134 +0000 2024-08-31T22:12:16.878 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2024-08-31T22:12:16.878 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-31T22:12:16.932 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-08-31T22:12:16.932 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-08-31T22:12:16.932 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000317814 s, 1.6 MB/s 2024-08-31T22:12:16.934 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-31T22:12:16.983 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2024-08-31T22:12:17.033 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-31T22:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-31T22:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 886 Links: 1 2024-08-31T22:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-31T22:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-08-31 22:06:35.545515631 +0000 2024-08-31T22:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-08-31 22:06:35.293517870 +0000 2024-08-31T22:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-08-31 22:06:35.293517870 +0000 2024-08-31T22:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2024-08-31T22:12:17.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-31T22:12:17.093 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-08-31T22:12:17.093 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-08-31T22:12:17.093 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.00021114 s, 2.4 MB/s 2024-08-31T22:12:17.094 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-31T22:12:17.142 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2024-08-31T22:12:17.189 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-31T22:12:17.189 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-31T22:12:17.189 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 900 Links: 1 2024-08-31T22:12:17.190 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-31T22:12:17.190 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-08-31 22:06:36.033511297 +0000 2024-08-31T22:12:17.190 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-08-31 22:06:35.773513607 +0000 2024-08-31T22:12:17.190 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-08-31 22:06:35.773513607 +0000 2024-08-31T22:12:17.190 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2024-08-31T22:12:17.190 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-31T22:12:17.243 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-08-31T22:12:17.243 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-08-31T22:12:17.243 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000276958 s, 1.8 MB/s 2024-08-31T22:12:17.245 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-31T22:12:17.294 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2024-08-31T22:12:17.341 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-31T22:12:17.341 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-31T22:12:17.341 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 918 Links: 1 2024-08-31T22:12:17.342 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-31T22:12:17.342 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-08-31 22:06:36.521506962 +0000 2024-08-31T22:12:17.342 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-08-31 22:06:36.245509413 +0000 2024-08-31T22:12:17.342 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-08-31 22:06:36.245509413 +0000 2024-08-31T22:12:17.342 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2024-08-31T22:12:17.342 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-31T22:12:17.396 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-08-31T22:12:17.396 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-08-31T22:12:17.396 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000296498 s, 1.7 MB/s 2024-08-31T22:12:17.398 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-31T22:12:17.447 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-31T22:12:17.447 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-08-31T22:12:17.447 INFO:tasks.ceph:Generating config... 2024-08-31T22:12:17.448 INFO:tasks.ceph:[client] debug rbd = 20 2024-08-31T22:12:17.448 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-08-31T22:12:17.448 INFO:tasks.ceph:[client] log to stderr = False 2024-08-31T22:12:17.448 INFO:tasks.ceph:[client] rbd default features = 125 2024-08-31T22:12:17.448 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.449 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.449 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.449 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.449 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.449 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.449 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.449 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.449 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.449 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.449 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.449 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.449 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.449 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.450 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-08-31T22:12:17.450 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-08-31T22:12:17.450 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-08-31T22:12:17.450 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-08-31T22:12:17.450 INFO:tasks.ceph:[mgr] debug ms = 1 2024-08-31T22:12:17.450 INFO:tasks.ceph:[mon] debug mon = 20 2024-08-31T22:12:17.450 INFO:tasks.ceph:[mon] debug ms = 1 2024-08-31T22:12:17.450 INFO:tasks.ceph:[mon] debug paxos = 20 2024-08-31T22:12:17.450 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-08-31T22:12:17.450 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-08-31T22:12:17.450 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-08-31T22:12:17.450 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-08-31T22:12:17.450 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-08-31T22:12:17.450 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-08-31T22:12:17.451 INFO:tasks.ceph:[osd] debug ms = 1 2024-08-31T22:12:17.451 INFO:tasks.ceph:[osd] debug osd = 20 2024-08-31T22:12:17.451 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-08-31T22:12:17.451 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-08-31T22:12:17.451 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-08-31T22:12:17.451 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-08-31T22:12:17.451 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-08-31T22:12:17.451 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-08-31T22:12:17.451 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-08-31T22:12:17.451 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-08-31T22:12:17.520 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/cluster1.keyring 2024-08-31T22:12:17.531 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-08-31T22:12:17.607 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-08-31T22:12:17.659 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.7')] 2024-08-31T22:12:17.660 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.7', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-08-31T22:12:17.660 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:17.661 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-08-31T22:12:17.704 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.7 --print /home/ubuntu/cephtest/cluster1.monmap 2024-08-31T22:12:17.779 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-08-31T22:12:17.780 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: generated fsid 8958e47a-d309-4bd5-ac60-bdf6f93061e5 2024-08-31T22:12:17.780 INFO:teuthology.orchestra.run.smithi007.stdout:setting min_mon_release = quincy 2024-08-31T22:12:17.780 INFO:teuthology.orchestra.run.smithi007.stdout:epoch 0 2024-08-31T22:12:17.780 INFO:teuthology.orchestra.run.smithi007.stdout:fsid 8958e47a-d309-4bd5-ac60-bdf6f93061e5 2024-08-31T22:12:17.780 INFO:teuthology.orchestra.run.smithi007.stdout:last_changed 2024-08-31T22:12:17.781296+0000 2024-08-31T22:12:17.780 INFO:teuthology.orchestra.run.smithi007.stdout:created 2024-08-31T22:12:17.781296+0000 2024-08-31T22:12:17.780 INFO:teuthology.orchestra.run.smithi007.stdout:min_mon_release 17 (quincy) 2024-08-31T22:12:17.780 INFO:teuthology.orchestra.run.smithi007.stdout:election_strategy: 1 2024-08-31T22:12:17.780 INFO:teuthology.orchestra.run.smithi007.stdout:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-08-31T22:12:17.780 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-08-31T22:12:17.781 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-08-31T22:12:17.829 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 8958e47a-d309-4bd5-ac60-bdf6f93061e5... 2024-08-31T22:12:17.830 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-08-31T22:12:17.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-08-31T22:12:17.903 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2024-08-31T22:12:17.903 INFO:teuthology.orchestra.run.smithi007.stdout: chdir = "" 2024-08-31T22:12:17.903 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-31T22:12:17.903 INFO:teuthology.orchestra.run.smithi007.stdout: auth supported = cephx 2024-08-31T22:12:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: filestore xattr use omap = true 2024-08-31T22:12:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: mon clock drift allowed = 1.000 2024-08-31T22:12:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: osd crush chooseleaf type = 0 2024-08-31T22:12:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: auth debug = true 2024-08-31T22:12:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on old message = true 2024-08-31T22:12:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on bug = true 2024-08-31T22:12:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-31T22:12:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: mon pg warn max object skew = 0 2024-08-31T22:12:17.904 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: # disable pg_autoscaler by default for new pools 2024-08-31T22:12:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-31T22:12:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default size = 2 2024-08-31T22:12:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow primary affinity = true 2024-08-31T22:12:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow pg remap = true 2024-08-31T22:12:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on legacy crush tunables = false 2024-08-31T22:12:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on crush straw calc version zero = false 2024-08-31T22:12:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on no sortbitwise = false 2024-08-31T22:12:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on osd down out interval zero = false 2024-08-31T22:12:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on too few osds = false 2024-08-31T22:12:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-31T22:12:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-31T22:12:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: mon_allow_pool_size_one = true 2024-08-31T22:12:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-31T22:12:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: osd default data pool replay window = 5 2024-08-31T22:12:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: mon allow pool delete = true 2024-08-31T22:12:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: mon cluster log file level = debug 2024-08-31T22:12:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: debug asserts on shutdown = true 2024-08-31T22:12:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: mon health detail to clog = false 2024-08-31T22:12:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: mon host = 172.21.15.7 2024-08-31T22:12:17.906 INFO:teuthology.orchestra.run.smithi007.stdout: mon client directed command retry = 5 2024-08-31T22:12:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on pool no app = False 2024-08-31T22:12:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject socket failures = 5000 2024-08-31T22:12:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = 8958e47a-d309-4bd5-ac60-bdf6f93061e5 2024-08-31T22:12:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.907 INFO:teuthology.orchestra.run.smithi007.stdout:[osd] 2024-08-31T22:12:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: osd journal size = 100 2024-08-31T22:12:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub load threshold = 5.0 2024-08-31T22:12:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub max interval = 600 2024-08-31T22:12:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock profile = high_recovery_ops 2024-08-31T22:12:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: osd recover clone overlap = true 2024-08-31T22:12:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: osd recovery max chunk = 1048576 2024-08-31T22:12:17.907 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug shutdown = true 2024-08-31T22:12:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug op order = true 2024-08-31T22:12:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify stray on activate = true 2024-08-31T22:12:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug trim objects = true 2024-08-31T22:12:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: osd open classes on start = true 2024-08-31T22:12:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug pg log writeout = true 2024-08-31T22:12:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: osd deep scrub update digest min age = 30 2024-08-31T22:12:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: osd map max advance = 10 2024-08-31T22:12:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.908 INFO:teuthology.orchestra.run.smithi007.stdout: journal zero on create = true 2024-08-31T22:12:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: filestore ondisk finisher threads = 3 2024-08-31T22:12:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: filestore apply finisher threads = 3 2024-08-31T22:12:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: bdev debug aio = true 2024-08-31T22:12:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug misdirected ops = true 2024-08-31T22:12:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore block size = 96636764160 2024-08-31T22:12:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore compression algorithm = zstd 2024-08-31T22:12:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore compression mode = aggressive 2024-08-31T22:12:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore fsck on mount = True 2024-08-31T22:12:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluefs = 1/20 2024-08-31T22:12:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluestore = 1/20 2024-08-31T22:12:17.909 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-08-31T22:12:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: debug osd = 20 2024-08-31T22:12:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: debug rocksdb = 4/10 2024-08-31T22:12:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd backfillfull_ratio = 0.85 2024-08-31T22:12:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd full ratio = 0.9 2024-08-31T22:12:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd nearfull ratio = 0.8 2024-08-31T22:12:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: osd failsafe full ratio = 0.95 2024-08-31T22:12:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: osd objectstore = bluestore 2024-08-31T22:12:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.910 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr] 2024-08-31T22:12:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-08-31T22:12:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: debug mgr = 20 2024-08-31T22:12:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2024-08-31T22:12:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-08-31T22:12:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-08-31T22:12:17.910 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-08-31T22:12:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: mgr/telemetry/nag = false 2024-08-31T22:12:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.911 INFO:teuthology.orchestra.run.smithi007.stdout:[mon] 2024-08-31T22:12:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-08-31T22:12:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2024-08-31T22:12:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: debug paxos = 20 2024-08-31T22:12:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-08-31T22:12:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: mon data avail warn = 5 2024-08-31T22:12:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: mon mgr mkfs grace = 240 2024-08-31T22:12:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-08-31T22:12:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd reporter subtree level = osd 2024-08-31T22:12:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd prime pg temp = true 2024-08-31T22:12:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-08-31T22:12:17.911 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-31T22:12:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: auth mon ticket ttl = 660 # 11m 2024-08-31T22:12:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: auth service ticket ttl = 240 # 4m 2024-08-31T22:12:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: # don't complain about insecure global_id in the test suite 2024-08-31T22:12:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-31T22:12:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-31T22:12:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: # 1m isn't quite enough 2024-08-31T22:12:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: mon_down_mkfs_grace = 2m 2024-08-31T22:12:17.912 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_filestore_osds = false 2024-08-31T22:12:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:17.913 INFO:teuthology.orchestra.run.smithi007.stdout:[client] 2024-08-31T22:12:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: rgw cache enabled = true 2024-08-31T22:12:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable ops log = true 2024-08-31T22:12:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable usage log = true 2024-08-31T22:12:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-31T22:12:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-31T22:12:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: debug rbd = 20 2024-08-31T22:12:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: debug rbd_mirror = 15 2024-08-31T22:12:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: log to stderr = False 2024-08-31T22:12:17.913 INFO:teuthology.orchestra.run.smithi007.stdout: rbd default features = 125 2024-08-31T22:12:17.914 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.a] 2024-08-31T22:12:17.914 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.0] 2024-08-31T22:12:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.914 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.1] 2024-08-31T22:12:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.914 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.2] 2024-08-31T22:12:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.914 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.3] 2024-08-31T22:12:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.914 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.914 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.4] 2024-08-31T22:12:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.915 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.5] 2024-08-31T22:12:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.915 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.6] 2024-08-31T22:12:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.915 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.916 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-08-31T22:12:17.916 INFO:teuthology.orchestra.run.smithi032.stdout: chdir = "" 2024-08-31T22:12:17.916 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-31T22:12:17.916 INFO:teuthology.orchestra.run.smithi032.stdout: auth supported = cephx 2024-08-31T22:12:17.916 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.916 INFO:teuthology.orchestra.run.smithi032.stdout: filestore xattr use omap = true 2024-08-31T22:12:17.916 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.916 INFO:teuthology.orchestra.run.smithi032.stdout: mon clock drift allowed = 1.000 2024-08-31T22:12:17.916 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.916 INFO:teuthology.orchestra.run.smithi032.stdout: osd crush chooseleaf type = 0 2024-08-31T22:12:17.916 INFO:teuthology.orchestra.run.smithi032.stdout: auth debug = true 2024-08-31T22:12:17.917 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.917 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on old message = true 2024-08-31T22:12:17.917 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on bug = true 2024-08-31T22:12:17.917 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.917 INFO:teuthology.orchestra.run.smithi032.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-31T22:12:17.917 INFO:teuthology.orchestra.run.smithi032.stdout: mon pg warn max object skew = 0 2024-08-31T22:12:17.917 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.917 INFO:teuthology.orchestra.run.smithi032.stdout: # disable pg_autoscaler by default for new pools 2024-08-31T22:12:17.917 INFO:teuthology.orchestra.run.smithi032.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-31T22:12:17.917 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.917 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default size = 2 2024-08-31T22:12:17.917 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.917 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow primary affinity = true 2024-08-31T22:12:17.918 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow pg remap = true 2024-08-31T22:12:17.918 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on legacy crush tunables = false 2024-08-31T22:12:17.918 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on crush straw calc version zero = false 2024-08-31T22:12:17.918 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on no sortbitwise = false 2024-08-31T22:12:17.918 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on osd down out interval zero = false 2024-08-31T22:12:17.918 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on too few osds = false 2024-08-31T22:12:17.918 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-31T22:12:17.918 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-31T22:12:17.918 INFO:teuthology.orchestra.run.smithi032.stdout: mon_allow_pool_size_one = true 2024-08-31T22:12:17.919 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.919 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-31T22:12:17.919 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.919 INFO:teuthology.orchestra.run.smithi032.stdout: osd default data pool replay window = 5 2024-08-31T22:12:17.919 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.919 INFO:teuthology.orchestra.run.smithi032.stdout: mon allow pool delete = true 2024-08-31T22:12:17.919 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.919 INFO:teuthology.orchestra.run.smithi032.stdout: mon cluster log file level = debug 2024-08-31T22:12:17.919 INFO:teuthology.orchestra.run.smithi032.stdout: debug asserts on shutdown = true 2024-08-31T22:12:17.919 INFO:teuthology.orchestra.run.smithi032.stdout: mon health detail to clog = false 2024-08-31T22:12:17.919 INFO:teuthology.orchestra.run.smithi032.stdout: mon host = 172.21.15.7 2024-08-31T22:12:17.919 INFO:teuthology.orchestra.run.smithi032.stdout: mon client directed command retry = 5 2024-08-31T22:12:17.919 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on pool no app = False 2024-08-31T22:12:17.920 INFO:teuthology.orchestra.run.smithi032.stdout: ms inject socket failures = 5000 2024-08-31T22:12:17.920 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = 8958e47a-d309-4bd5-ac60-bdf6f93061e5 2024-08-31T22:12:17.920 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.920 INFO:teuthology.orchestra.run.smithi032.stdout:[osd] 2024-08-31T22:12:17.920 INFO:teuthology.orchestra.run.smithi032.stdout: osd journal size = 100 2024-08-31T22:12:17.920 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.920 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub load threshold = 5.0 2024-08-31T22:12:17.920 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub max interval = 600 2024-08-31T22:12:17.920 INFO:teuthology.orchestra.run.smithi032.stdout: osd mclock profile = high_recovery_ops 2024-08-31T22:12:17.921 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.921 INFO:teuthology.orchestra.run.smithi032.stdout: osd recover clone overlap = true 2024-08-31T22:12:17.921 INFO:teuthology.orchestra.run.smithi032.stdout: osd recovery max chunk = 1048576 2024-08-31T22:12:17.921 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.921 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug shutdown = true 2024-08-31T22:12:17.921 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug op order = true 2024-08-31T22:12:17.921 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify stray on activate = true 2024-08-31T22:12:17.921 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug trim objects = true 2024-08-31T22:12:17.921 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.921 INFO:teuthology.orchestra.run.smithi032.stdout: osd open classes on start = true 2024-08-31T22:12:17.921 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug pg log writeout = true 2024-08-31T22:12:17.921 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.921 INFO:teuthology.orchestra.run.smithi032.stdout: osd deep scrub update digest min age = 30 2024-08-31T22:12:17.922 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.922 INFO:teuthology.orchestra.run.smithi032.stdout: osd map max advance = 10 2024-08-31T22:12:17.922 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.922 INFO:teuthology.orchestra.run.smithi032.stdout: journal zero on create = true 2024-08-31T22:12:17.922 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.922 INFO:teuthology.orchestra.run.smithi032.stdout: filestore ondisk finisher threads = 3 2024-08-31T22:12:17.922 INFO:teuthology.orchestra.run.smithi032.stdout: filestore apply finisher threads = 3 2024-08-31T22:12:17.922 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.922 INFO:teuthology.orchestra.run.smithi032.stdout: bdev debug aio = true 2024-08-31T22:12:17.922 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug misdirected ops = true 2024-08-31T22:12:17.922 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore block size = 96636764160 2024-08-31T22:12:17.923 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore compression algorithm = zstd 2024-08-31T22:12:17.923 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore compression mode = aggressive 2024-08-31T22:12:17.923 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore fsck on mount = True 2024-08-31T22:12:17.923 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluefs = 1/20 2024-08-31T22:12:17.923 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluestore = 1/20 2024-08-31T22:12:17.923 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-08-31T22:12:17.923 INFO:teuthology.orchestra.run.smithi032.stdout: debug osd = 20 2024-08-31T22:12:17.923 INFO:teuthology.orchestra.run.smithi032.stdout: debug rocksdb = 4/10 2024-08-31T22:12:17.923 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd backfillfull_ratio = 0.85 2024-08-31T22:12:17.923 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd full ratio = 0.9 2024-08-31T22:12:17.923 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd nearfull ratio = 0.8 2024-08-31T22:12:17.923 INFO:teuthology.orchestra.run.smithi032.stdout: osd failsafe full ratio = 0.95 2024-08-31T22:12:17.923 INFO:teuthology.orchestra.run.smithi032.stdout: osd objectstore = bluestore 2024-08-31T22:12:17.923 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.924 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr] 2024-08-31T22:12:17.924 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-08-31T22:12:17.924 INFO:teuthology.orchestra.run.smithi032.stdout: debug mgr = 20 2024-08-31T22:12:17.924 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-08-31T22:12:17.924 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-08-31T22:12:17.924 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-08-31T22:12:17.924 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-08-31T22:12:17.924 INFO:teuthology.orchestra.run.smithi032.stdout: mgr/telemetry/nag = false 2024-08-31T22:12:17.924 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.924 INFO:teuthology.orchestra.run.smithi032.stdout:[mon] 2024-08-31T22:12:17.925 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-08-31T22:12:17.925 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-08-31T22:12:17.925 INFO:teuthology.orchestra.run.smithi032.stdout: debug paxos = 20 2024-08-31T22:12:17.925 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-08-31T22:12:17.925 INFO:teuthology.orchestra.run.smithi032.stdout: mon data avail warn = 5 2024-08-31T22:12:17.925 INFO:teuthology.orchestra.run.smithi032.stdout: mon mgr mkfs grace = 240 2024-08-31T22:12:17.925 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-08-31T22:12:17.925 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd reporter subtree level = osd 2024-08-31T22:12:17.925 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd prime pg temp = true 2024-08-31T22:12:17.925 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-08-31T22:12:17.925 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.925 INFO:teuthology.orchestra.run.smithi032.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-31T22:12:17.925 INFO:teuthology.orchestra.run.smithi032.stdout: auth mon ticket ttl = 660 # 11m 2024-08-31T22:12:17.925 INFO:teuthology.orchestra.run.smithi032.stdout: auth service ticket ttl = 240 # 4m 2024-08-31T22:12:17.926 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.926 INFO:teuthology.orchestra.run.smithi032.stdout: # don't complain about insecure global_id in the test suite 2024-08-31T22:12:17.926 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-31T22:12:17.926 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-31T22:12:17.926 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.926 INFO:teuthology.orchestra.run.smithi032.stdout: # 1m isn't quite enough 2024-08-31T22:12:17.926 INFO:teuthology.orchestra.run.smithi032.stdout: mon_down_mkfs_grace = 2m 2024-08-31T22:12:17.926 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.926 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_filestore_osds = false 2024-08-31T22:12:17.926 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:17.926 INFO:teuthology.orchestra.run.smithi032.stdout:[client] 2024-08-31T22:12:17.926 INFO:teuthology.orchestra.run.smithi032.stdout: rgw cache enabled = true 2024-08-31T22:12:17.927 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable ops log = true 2024-08-31T22:12:17.927 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable usage log = true 2024-08-31T22:12:17.927 INFO:teuthology.orchestra.run.smithi032.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-31T22:12:17.927 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-31T22:12:17.927 INFO:teuthology.orchestra.run.smithi032.stdout: debug rbd = 20 2024-08-31T22:12:17.927 INFO:teuthology.orchestra.run.smithi032.stdout: debug rbd_mirror = 15 2024-08-31T22:12:17.927 INFO:teuthology.orchestra.run.smithi032.stdout: log to stderr = False 2024-08-31T22:12:17.927 INFO:teuthology.orchestra.run.smithi032.stdout: rbd default features = 125 2024-08-31T22:12:17.927 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.a] 2024-08-31T22:12:17.927 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.0] 2024-08-31T22:12:17.927 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.927 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.927 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.1] 2024-08-31T22:12:17.927 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.928 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.928 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.2] 2024-08-31T22:12:17.928 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.928 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.928 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.3] 2024-08-31T22:12:17.928 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.928 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.928 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.4] 2024-08-31T22:12:17.928 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.928 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.928 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.5] 2024-08-31T22:12:17.928 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.928 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.929 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.6] 2024-08-31T22:12:17.929 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:17.929 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:17.930 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-08-31T22:12:17.931 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-08-31T22:12:18.023 INFO:tasks.ceph:Copying monmap to all nodes... 2024-08-31T22:12:18.023 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:18.023 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-08-31T22:12:18.029 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:18.029 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-08-31T22:12:18.077 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi007.front.sepia.ceph.com 2024-08-31T22:12:18.077 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:18.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/cluster1.keyring 2024-08-31T22:12:18.078 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-08-31T22:12:18.142 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:18.142 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-08-31T22:12:18.190 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi032.front.sepia.ceph.com 2024-08-31T22:12:18.190 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:18.190 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/cluster1.keyring 2024-08-31T22:12:18.190 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-08-31T22:12:18.214 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:18.214 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-08-31T22:12:18.262 INFO:tasks.ceph:Setting up mon nodes... 2024-08-31T22:12:18.262 INFO:tasks.ceph:Setting up mgr nodes... 2024-08-31T22:12:18.262 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-08-31T22:12:18.307 INFO:teuthology.orchestra.run.smithi007.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-08-31T22:12:18.310 INFO:tasks.ceph:Setting up mds nodes... 2024-08-31T22:12:18.310 INFO:tasks.ceph_client:Setting up client nodes... 2024-08-31T22:12:18.311 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-08-31T22:12:18.392 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-08-31T22:12:18.404 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-08-31T22:12:18.440 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-08-31T22:12:18.460 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-08-31T22:12:18.545 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-08-31T22:12:18.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-08-31T22:12:18.628 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-08-31T22:12:18.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-08-31T22:12:18.716 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-08-31T22:12:18.725 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-08-31T22:12:18.798 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-08-31T22:12:18.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-08-31T22:12:18.891 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-08-31T22:12:18.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-08-31T22:12:18.968 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-08-31T22:12:18.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-08-31T22:12:19.057 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-08-31T22:12:19.069 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-08-31T22:12:19.069 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-08-31T22:12:19.069 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-08-31T22:12:19.083 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-31T22:12:19.084 INFO:tasks.ceph:role: cluster1.osd.0 2024-08-31T22:12:19.084 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com 2024-08-31T22:12:19.084 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-31T22:12:19.137 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-31T22:12:19.181 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-31T22:12:19.181 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-31T22:12:19.181 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-08-31T22:12:19.181 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-31T22:12:19.181 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-08-31T22:12:19.181 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-31T22:12:19.181 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-08-31T22:12:19.182 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-31T22:12:19.182 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-31T22:12:19.183 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-08-31T22:12:19.183 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-08-31T22:12:19.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-08-31T22:12:19.241 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-08-31T22:12:19.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T22:12:19.294 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: /sbin/restorecon: command not found 2024-08-31T22:12:19.294 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-08-31T22:12:19.348 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-31T22:12:19.348 INFO:tasks.ceph:role: cluster1.osd.1 2024-08-31T22:12:19.348 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com 2024-08-31T22:12:19.348 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-31T22:12:19.401 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-31T22:12:19.401 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-31T22:12:19.401 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-31T22:12:19.401 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-08-31T22:12:19.402 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-31T22:12:19.402 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-08-31T22:12:19.402 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-31T22:12:19.402 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-08-31T22:12:19.402 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-31T22:12:19.402 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-31T22:12:19.424 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-08-31T22:12:19.427 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-08-31T22:12:19.427 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-08-31T22:12:19.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-08-31T22:12:19.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T22:12:19.542 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: /sbin/restorecon: command not found 2024-08-31T22:12:19.543 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-08-31T22:12:19.595 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-31T22:12:19.596 INFO:tasks.ceph:role: cluster1.osd.2 2024-08-31T22:12:19.596 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com 2024-08-31T22:12:19.596 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-31T22:12:19.651 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-31T22:12:19.651 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-31T22:12:19.651 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-31T22:12:19.651 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-08-31T22:12:19.651 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-31T22:12:19.652 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-08-31T22:12:19.652 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-31T22:12:19.652 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-08-31T22:12:19.652 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-31T22:12:19.652 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-31T22:12:19.675 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-08-31T22:12:19.677 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-08-31T22:12:19.677 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-08-31T22:12:19.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-08-31T22:12:19.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T22:12:19.790 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: /sbin/restorecon: command not found 2024-08-31T22:12:19.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-08-31T22:12:19.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-08-31T22:12:19.894+0000 7f5b62a477c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-08-31T22:12:19.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-08-31T22:12:19.894+0000 7f5b62a477c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-08-31T22:12:19.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-08-31T22:12:19.894+0000 7f5b62a477c0 -1 bdev(0x55870cfac380 /var/lib/ceph/osd/cluster1-0/block) open stat got: (1) Operation not permitted 2024-08-31T22:12:19.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-08-31T22:12:19.894+0000 7f5b62a477c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-08-31T22:12:23.059 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-08-31T22:12:23.074 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-08-31T22:12:23.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-08-31T22:12:23.150+0000 7f504ebe97c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-08-31T22:12:23.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-08-31T22:12:23.154+0000 7f504ebe97c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-08-31T22:12:23.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-08-31T22:12:23.154+0000 7f504ebe97c0 -1 bdev(0x55b811068380 /var/lib/ceph/osd/cluster1-1/block) open stat got: (1) Operation not permitted 2024-08-31T22:12:23.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-08-31T22:12:23.154+0000 7f504ebe97c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-08-31T22:12:26.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-08-31T22:12:26.371 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-08-31T22:12:26.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-08-31T22:12:26.450+0000 7f702cb607c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-08-31T22:12:26.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-08-31T22:12:26.450+0000 7f702cb607c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-08-31T22:12:26.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-08-31T22:12:26.450+0000 7f702cb607c0 -1 bdev(0x563b0a1d6380 /var/lib/ceph/osd/cluster1-2/block) open stat got: (1) Operation not permitted 2024-08-31T22:12:26.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-08-31T22:12:26.450+0000 7f702cb607c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-08-31T22:12:29.575 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-08-31T22:12:29.630 INFO:tasks.ceph:Reading keys from all nodes... 2024-08-31T22:12:29.630 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:29.630 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-08-31T22:12:29.684 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:29.684 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-08-31T22:12:29.745 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:29.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-08-31T22:12:29.800 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:29.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-08-31T22:12:29.856 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:29.856 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-08-31T22:12:29.906 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:29.906 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-08-31T22:12:29.912 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:29.912 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-08-31T22:12:29.962 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:29.962 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-08-31T22:12:30.010 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:30.010 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-08-31T22:12:30.058 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:30.058 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-08-31T22:12:30.106 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:30.106 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-08-31T22:12:30.154 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:30.154 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-08-31T22:12:30.201 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:30.202 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-08-31T22:12:30.249 INFO:tasks.ceph:Adding keys to all mons... 2024-08-31T22:12:30.250 DEBUG:teuthology.orchestra.run.smithi007:> sudo tee -a /etc/ceph/cluster1.keyring 2024-08-31T22:12:30.263 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.x] 2024-08-31T22:12:30.263 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBCldNmi9VpEhAAdX+TJ0O0Xt8anvEEnTAFGA== 2024-08-31T22:12:30.264 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.0] 2024-08-31T22:12:30.264 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBDldNmo8OfNRAArtLds+dwQm5NqS98lEzpuQ== 2024-08-31T22:12:30.264 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.1] 2024-08-31T22:12:30.264 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBHldNm4UVsCRAAexwN/445XHexiA4S84koFg== 2024-08-31T22:12:30.264 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.2] 2024-08-31T22:12:30.264 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBKldNmD1YcGxAAECqQ8HgUBUEryXKCRHaS9g== 2024-08-31T22:12:30.264 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2024-08-31T22:12:30.264 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBCldNmpll4FxAANCZYMjywXk9ZENdYFIWfWw== 2024-08-31T22:12:30.264 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror] 2024-08-31T22:12:30.264 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBCldNmZvZgGhAABvbh6KrOP92vspWT+djHFA== 2024-08-31T22:12:30.264 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.0] 2024-08-31T22:12:30.265 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBCldNmIRaZIBAA5D/O78xVRqMaWq5i1ALKDg== 2024-08-31T22:12:30.265 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.1] 2024-08-31T22:12:30.265 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBCldNmdN2SJRAAqIIJXHJsL8PYPso06bmeJw== 2024-08-31T22:12:30.265 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.2] 2024-08-31T22:12:30.265 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBCldNmUUTJKhAAtWLMhzIEEgMrZfO2RR1cPw== 2024-08-31T22:12:30.265 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.3] 2024-08-31T22:12:30.265 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBCldNm4QOsLxAAYycFBGOT5qMylr2tDf5gVQ== 2024-08-31T22:12:30.265 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.4] 2024-08-31T22:12:30.265 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBCldNmhPI5NRAAZhTZqRhPHUOCsvo7eY7cqg== 2024-08-31T22:12:30.265 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.5] 2024-08-31T22:12:30.265 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBCldNmNLbPORAAPCiMrYgb4ZFp8I+ccMR4Dg== 2024-08-31T22:12:30.265 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.6] 2024-08-31T22:12:30.265 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBDldNm6gOKAxAAHZ8F3h9oEm7xqCozbtMpvg== 2024-08-31T22:12:30.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-31T22:12:30.355 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T22:12:30.397 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T22:12:30.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T22:12:30.528 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T22:12:30.619 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T22:12:30.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T22:12:30.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T22:12:30.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T22:12:30.847 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T22:12:30.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T22:12:31.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T22:12:31.065 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T22:12:31.156 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-08-31T22:12:31.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-08-31T22:12:31.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-08-31T22:12:31.305 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-08-31T22:12:31.315 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-08-31T22:12:31.360 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-08-31T22:12:31.360 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-08-31T22:12:31.360 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-08-31T22:12:31.401 INFO:tasks.ceph.cluster1.mon.a:Started 2024-08-31T22:12:31.402 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-08-31T22:12:31.402 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-08-31T22:12:31.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-08-31T22:12:31.404 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-08-31T22:12:31.404 INFO:tasks.ceph:Setting crush tunables to default 2024-08-31T22:12:31.404 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster cluster1 osd crush tunables default 2024-08-31T22:12:31.615 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2024-08-31T22:12:31.633 INFO:tasks.ceph:check_enable_crimson: False 2024-08-31T22:12:31.633 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-08-31T22:12:31.633 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:31.634 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-08-31T22:12:31.645 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:31.646 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-08-31T22:12:31.704 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:31.704 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-08-31T22:12:31.760 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster cluster1 osd new e7308cb5-1737-4816-a3fb-c3ed18553f6b 0 2024-08-31T22:12:31.782 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:31.778+0000 7effd48e9180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-31T22:12:31.990 INFO:teuthology.orchestra.run.smithi007.stdout:0 2024-08-31T22:12:32.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster cluster1 osd new ce69f125-bfb1-463d-97f5-c2aa67f0124f 1 2024-08-31T22:12:32.207 INFO:teuthology.orchestra.run.smithi007.stdout:1 2024-08-31T22:12:32.224 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster cluster1 osd new f67f0693-4771-4a20-98a1-d2b6108a1ee0 2 2024-08-31T22:12:32.290 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:32.286+0000 7effd48e9180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-31T22:12:32.429 INFO:teuthology.orchestra.run.smithi007.stdout:2 2024-08-31T22:12:32.445 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-08-31T22:12:32.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-08-31T22:12:32.447 INFO:tasks.ceph.cluster1.osd.0:Started 2024-08-31T22:12:32.448 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-08-31T22:12:32.448 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-08-31T22:12:32.450 INFO:tasks.ceph.cluster1.osd.1:Started 2024-08-31T22:12:32.450 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-08-31T22:12:32.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-08-31T22:12:32.452 INFO:tasks.ceph.cluster1.osd.2:Started 2024-08-31T22:12:32.453 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-08-31T22:12:32.502 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:32.498+0000 7effd48e9180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-31T22:12:32.635 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:32.635 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":5,"fsid":"8958e47a-d309-4bd5-ac60-bdf6f93061e5","created":"2024-08-31T22:12:31.498952+0000","modified":"2024-08-31T22:12:32.429991+0000","last_up_change":"0.000000","last_in_change":"2024-08-31T22:12:32.429991+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7308cb5-1737-4816-a3fb-c3ed18553f6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ce69f125-bfb1-463d-97f5-c2aa67f0124f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f67f0693-4771-4a20-98a1-d2b6108a1ee0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T22:12:32.650 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-08-31T22:12:32.650 INFO:tasks.ceph:Waiting for OSDs to come up 2024-08-31T22:12:32.953 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-08-31T22:12:33.091 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:33.086+0000 7effd48e9180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-31T22:12:33.138 INFO:teuthology.misc.health.smithi007.stdout: 2024-08-31T22:12:33.138 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":5,"fsid":"8958e47a-d309-4bd5-ac60-bdf6f93061e5","created":"2024-08-31T22:12:31.498952+0000","modified":"2024-08-31T22:12:32.429991+0000","last_up_change":"0.000000","last_in_change":"2024-08-31T22:12:32.429991+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7308cb5-1737-4816-a3fb-c3ed18553f6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ce69f125-bfb1-463d-97f5-c2aa67f0124f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f67f0693-4771-4a20-98a1-d2b6108a1ee0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T22:12:33.148 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:33.146+0000 7effd48e9180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-31T22:12:33.154 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-08-31T22:12:33.204 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:33.198+0000 7effd48e9180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-31T22:12:33.285 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:33.282+0000 7effd48e9180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-31T22:12:33.339 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:33.334+0000 7effd48e9180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-31T22:12:33.419 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:33.414+0000 7effd48e9180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-31T22:12:33.475 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:33.470+0000 7effd48e9180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-31T22:12:33.633 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:33.630+0000 7effd48e9180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-31T22:12:33.697 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:33.694+0000 7effd48e9180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-31T22:12:34.089 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:12:34.086+0000 7f65995ea7c0 -1 Falling back to public interface 2024-08-31T22:12:34.105 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:12:34.102+0000 7f339b2197c0 -1 Falling back to public interface 2024-08-31T22:12:34.121 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:12:34.118+0000 7f1a881b97c0 -1 Falling back to public interface 2024-08-31T22:12:34.342 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-08-31T22:12:34.342 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-08-31T22:12:34.342 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr: from numpy import show_config as show_numpy_config 2024-08-31T22:12:34.404 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:34.398+0000 7effd48e9180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-31T22:12:34.547 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:34.542+0000 7effd48e9180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-31T22:12:34.727 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:34.722+0000 7effd48e9180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-31T22:12:34.795 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:34.790+0000 7effd48e9180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-31T22:12:34.936 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:34.934+0000 7effd48e9180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-31T22:12:35.566 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:35.562+0000 7effd48e9180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-31T22:12:35.645 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:35.642+0000 7effd48e9180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-31T22:12:35.835 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:35.830+0000 7effd48e9180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-31T22:12:35.912 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:35.906+0000 7effd48e9180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-31T22:12:36.217 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:36.214+0000 7effd48e9180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-31T22:12:36.282 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:36.278+0000 7effd48e9180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-31T22:12:37.515 INFO:tasks.ceph.cluster1.mgr.x.smithi007.stderr:2024-08-31T22:12:37.510+0000 7eff7b420640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-08-31T22:12:38.073 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:12:38.070+0000 7f65995ea7c0 -1 osd.0 0 log_to_monitors true 2024-08-31T22:12:38.191 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:12:38.186+0000 7f339b2197c0 -1 osd.2 0 log_to_monitors true 2024-08-31T22:12:38.432 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:12:38.426+0000 7f1a881b97c0 -1 osd.1 0 log_to_monitors true 2024-08-31T22:12:39.457 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-08-31T22:12:39.930 INFO:teuthology.misc.health.smithi007.stdout: 2024-08-31T22:12:39.930 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":7,"fsid":"8958e47a-d309-4bd5-ac60-bdf6f93061e5","created":"2024-08-31T22:12:31.498952+0000","modified":"2024-08-31T22:12:39.312429+0000","last_up_change":"0.000000","last_in_change":"2024-08-31T22:12:32.429991+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7308cb5-1737-4816-a3fb-c3ed18553f6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ce69f125-bfb1-463d-97f5-c2aa67f0124f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f67f0693-4771-4a20-98a1-d2b6108a1ee0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T22:12:39.954 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-08-31T22:12:40.267 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:12:40.262+0000 7f6595da5640 -1 osd.0 0 waiting for initial osdmap 2024-08-31T22:12:40.285 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:12:40.282+0000 7f658d3c8640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-31T22:12:40.302 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:12:40.298+0000 7f33971c2640 -1 osd.2 0 waiting for initial osdmap 2024-08-31T22:12:40.329 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:12:40.326+0000 7f338eff7640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-31T22:12:41.132 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:12:41.126+0000 7f1a84162640 -1 osd.1 0 waiting for initial osdmap 2024-08-31T22:12:41.157 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:12:41.154+0000 7f1a7bf97640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-31T22:12:46.257 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-08-31T22:12:46.564 INFO:teuthology.misc.health.smithi007.stdout: 2024-08-31T22:12:46.564 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":13,"fsid":"8958e47a-d309-4bd5-ac60-bdf6f93061e5","created":"2024-08-31T22:12:31.498952+0000","modified":"2024-08-31T22:12:45.351917+0000","last_up_change":"2024-08-31T22:12:41.323775+0000","last_in_change":"2024-08-31T22:12:32.429991+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-31T22:12:42.384098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7308cb5-1737-4816-a3fb-c3ed18553f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2244136370},{"type":"v1","addr":"172.21.15.7:6801","nonce":2244136370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2244136370},{"type":"v1","addr":"172.21.15.7:6803","nonce":2244136370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2244136370},{"type":"v1","addr":"172.21.15.7:6807","nonce":2244136370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2244136370},{"type":"v1","addr":"172.21.15.7:6805","nonce":2244136370}]},"public_addr":"172.21.15.7:6801/2244136370","cluster_addr":"172.21.15.7:6803/2244136370","heartbeat_back_addr":"172.21.15.7:6807/2244136370","heartbeat_front_addr":"172.21.15.7:6805/2244136370","state":["exists","up"]},{"osd":1,"uuid":"ce69f125-bfb1-463d-97f5-c2aa67f0124f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2889407906},{"type":"v1","addr":"172.21.15.7:6817","nonce":2889407906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":2889407906},{"type":"v1","addr":"172.21.15.7:6819","nonce":2889407906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":2889407906},{"type":"v1","addr":"172.21.15.7:6823","nonce":2889407906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":2889407906},{"type":"v1","addr":"172.21.15.7:6821","nonce":2889407906}]},"public_addr":"172.21.15.7:6817/2889407906","cluster_addr":"172.21.15.7:6819/2889407906","heartbeat_back_addr":"172.21.15.7:6823/2889407906","heartbeat_front_addr":"172.21.15.7:6821/2889407906","state":["exists","up"]},{"osd":2,"uuid":"f67f0693-4771-4a20-98a1-d2b6108a1ee0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2788741735},{"type":"v1","addr":"172.21.15.7:6809","nonce":2788741735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2788741735},{"type":"v1","addr":"172.21.15.7:6811","nonce":2788741735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2788741735},{"type":"v1","addr":"172.21.15.7:6815","nonce":2788741735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2788741735},{"type":"v1","addr":"172.21.15.7:6813","nonce":2788741735}]},"public_addr":"172.21.15.7:6809/2788741735","cluster_addr":"172.21.15.7:6811/2788741735","heartbeat_back_addr":"172.21.15.7:6815/2788741735","heartbeat_front_addr":"172.21.15.7:6813/2788741735","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:12:39.036879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:12:39.404760+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:12:39.188110+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T22:12:46.582 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-08-31T22:12:46.582 INFO:tasks.ceph:Creating RBD pool 2024-08-31T22:12:46.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-08-31T22:12:47.372 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'rbd' created 2024-08-31T22:12:47.401 DEBUG:teuthology.orchestra.run.smithi007:> rbd --cluster cluster1 pool init rbd 2024-08-31T22:12:50.417 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-08-31T22:12:50.417 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-08-31T22:12:50.417 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-08-31T22:12:50.418 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-08-31T22:12:50.418 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-08-31T22:12:50.773 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:50.789 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":784813447},{"type":"v1","addr":"172.21.15.7:6825","nonce":784813447}]},"active_addr":"172.21.15.7:6825/784813447","active_change":"2024-08-31T22:12:36.303485+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":1111074154}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":722725930}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":756028496}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":87235339}]}]} 2024-08-31T22:12:50.791 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-08-31T22:12:50.791 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-08-31T22:12:50.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-08-31T22:12:51.096 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:51.096 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":17,"fsid":"8958e47a-d309-4bd5-ac60-bdf6f93061e5","created":"2024-08-31T22:12:31.498952+0000","modified":"2024-08-31T22:12:50.384888+0000","last_up_change":"2024-08-31T22:12:41.323775+0000","last_in_change":"2024-08-31T22:12:32.429991+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-31T22:12:42.384098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-31T22:12:46.929549+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7308cb5-1737-4816-a3fb-c3ed18553f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2244136370},{"type":"v1","addr":"172.21.15.7:6801","nonce":2244136370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2244136370},{"type":"v1","addr":"172.21.15.7:6803","nonce":2244136370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2244136370},{"type":"v1","addr":"172.21.15.7:6807","nonce":2244136370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2244136370},{"type":"v1","addr":"172.21.15.7:6805","nonce":2244136370}]},"public_addr":"172.21.15.7:6801/2244136370","cluster_addr":"172.21.15.7:6803/2244136370","heartbeat_back_addr":"172.21.15.7:6807/2244136370","heartbeat_front_addr":"172.21.15.7:6805/2244136370","state":["exists","up"]},{"osd":1,"uuid":"ce69f125-bfb1-463d-97f5-c2aa67f0124f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2889407906},{"type":"v1","addr":"172.21.15.7:6817","nonce":2889407906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":2889407906},{"type":"v1","addr":"172.21.15.7:6819","nonce":2889407906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":2889407906},{"type":"v1","addr":"172.21.15.7:6823","nonce":2889407906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":2889407906},{"type":"v1","addr":"172.21.15.7:6821","nonce":2889407906}]},"public_addr":"172.21.15.7:6817/2889407906","cluster_addr":"172.21.15.7:6819/2889407906","heartbeat_back_addr":"172.21.15.7:6823/2889407906","heartbeat_front_addr":"172.21.15.7:6821/2889407906","state":["exists","up"]},{"osd":2,"uuid":"f67f0693-4771-4a20-98a1-d2b6108a1ee0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2788741735},{"type":"v1","addr":"172.21.15.7:6809","nonce":2788741735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2788741735},{"type":"v1","addr":"172.21.15.7:6811","nonce":2788741735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2788741735},{"type":"v1","addr":"172.21.15.7:6815","nonce":2788741735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2788741735},{"type":"v1","addr":"172.21.15.7:6813","nonce":2788741735}]},"public_addr":"172.21.15.7:6809/2788741735","cluster_addr":"172.21.15.7:6811/2788741735","heartbeat_back_addr":"172.21.15.7:6815/2788741735","heartbeat_front_addr":"172.21.15.7:6813/2788741735","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:12:39.036879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:12:39.404760+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:12:39.188110+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T22:12:51.115 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-08-31T22:12:51.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-08-31T22:12:51.413 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:51.413 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":17,"fsid":"8958e47a-d309-4bd5-ac60-bdf6f93061e5","created":"2024-08-31T22:12:31.498952+0000","modified":"2024-08-31T22:12:50.384888+0000","last_up_change":"2024-08-31T22:12:41.323775+0000","last_in_change":"2024-08-31T22:12:32.429991+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-31T22:12:42.384098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-31T22:12:46.929549+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7308cb5-1737-4816-a3fb-c3ed18553f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2244136370},{"type":"v1","addr":"172.21.15.7:6801","nonce":2244136370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2244136370},{"type":"v1","addr":"172.21.15.7:6803","nonce":2244136370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2244136370},{"type":"v1","addr":"172.21.15.7:6807","nonce":2244136370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2244136370},{"type":"v1","addr":"172.21.15.7:6805","nonce":2244136370}]},"public_addr":"172.21.15.7:6801/2244136370","cluster_addr":"172.21.15.7:6803/2244136370","heartbeat_back_addr":"172.21.15.7:6807/2244136370","heartbeat_front_addr":"172.21.15.7:6805/2244136370","state":["exists","up"]},{"osd":1,"uuid":"ce69f125-bfb1-463d-97f5-c2aa67f0124f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2889407906},{"type":"v1","addr":"172.21.15.7:6817","nonce":2889407906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":2889407906},{"type":"v1","addr":"172.21.15.7:6819","nonce":2889407906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":2889407906},{"type":"v1","addr":"172.21.15.7:6823","nonce":2889407906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":2889407906},{"type":"v1","addr":"172.21.15.7:6821","nonce":2889407906}]},"public_addr":"172.21.15.7:6817/2889407906","cluster_addr":"172.21.15.7:6819/2889407906","heartbeat_back_addr":"172.21.15.7:6823/2889407906","heartbeat_front_addr":"172.21.15.7:6821/2889407906","state":["exists","up"]},{"osd":2,"uuid":"f67f0693-4771-4a20-98a1-d2b6108a1ee0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2788741735},{"type":"v1","addr":"172.21.15.7:6809","nonce":2788741735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2788741735},{"type":"v1","addr":"172.21.15.7:6811","nonce":2788741735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2788741735},{"type":"v1","addr":"172.21.15.7:6815","nonce":2788741735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2788741735},{"type":"v1","addr":"172.21.15.7:6813","nonce":2788741735}]},"public_addr":"172.21.15.7:6809/2788741735","cluster_addr":"172.21.15.7:6811/2788741735","heartbeat_back_addr":"172.21.15.7:6815/2788741735","heartbeat_front_addr":"172.21.15.7:6813/2788741735","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:12:39.036879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:12:39.404760+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:12:39.188110+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T22:12:51.433 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-08-31T22:12:51.433 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-08-31T22:12:51.434 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-08-31T22:12:51.599 INFO:teuthology.orchestra.run.smithi007.stdout:34359738374 2024-08-31T22:12:51.599 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-08-31T22:12:51.605 INFO:teuthology.orchestra.run.smithi007.stdout:38654705668 2024-08-31T22:12:51.605 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-08-31T22:12:51.606 INFO:teuthology.orchestra.run.smithi007.stdout:38654705668 2024-08-31T22:12:51.606 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-08-31T22:12:51.906 INFO:teuthology.orchestra.run.smithi007.stdout:34359738372 2024-08-31T22:12:51.923 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738372 for osd.0 2024-08-31T22:12:51.937 INFO:teuthology.orchestra.run.smithi007.stdout:38654705667 2024-08-31T22:12:51.952 INFO:teuthology.orchestra.run.smithi007.stdout:38654705667 2024-08-31T22:12:51.954 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705668 got 38654705667 for osd.1 2024-08-31T22:12:51.969 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705668 got 38654705667 for osd.2 2024-08-31T22:12:52.924 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-08-31T22:12:52.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-08-31T22:12:52.970 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-08-31T22:12:53.237 INFO:teuthology.orchestra.run.smithi007.stdout:34359738374 2024-08-31T22:12:53.254 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.0 2024-08-31T22:12:53.254 DEBUG:teuthology.parallel:result is None 2024-08-31T22:12:53.296 INFO:teuthology.orchestra.run.smithi007.stdout:38654705668 2024-08-31T22:12:53.313 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705668 got 38654705668 for osd.1 2024-08-31T22:12:53.313 DEBUG:teuthology.parallel:result is None 2024-08-31T22:12:53.328 INFO:teuthology.orchestra.run.smithi007.stdout:38654705668 2024-08-31T22:12:53.345 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705668 got 38654705668 for osd.2 2024-08-31T22:12:53.345 DEBUG:teuthology.parallel:result is None 2024-08-31T22:12:53.345 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-08-31T22:12:53.345 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-08-31T22:12:53.678 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:53.678 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-08-31T22:12:53.696 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-08-31T22:12:52.313280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":94120,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283021400,"statfs":{"total":289910292480,"available":289813913600,"internally_reserved":0,"allocated":557056,"data_stored":1306854,"data_compressed":6360,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.939375"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:12:50.394482+0000","last_change":"2024-08-31T22:12:50.394482+0000","last_active":"2024-08-31T22:12:50.394482+0000","last_peered":"2024-08-31T22:12:50.394482+0000","last_clean":"2024-08-31T22:12:50.394482+0000","last_became_active":"2024-08-31T22:12:48.393530+0000","last_became_peered":"2024-08-31T22:12:48.393530+0000","last_unstale":"2024-08-31T22:12:50.394482+0000","last_undegraded":"2024-08-31T22:12:50.394482+0000","last_fullsized":"2024-08-31T22:12:50.394482+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:53:59.887015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021475320000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:12:50.394918+0000","last_change":"2024-08-31T22:12:50.394918+0000","last_active":"2024-08-31T22:12:50.394918+0000","last_peered":"2024-08-31T22:12:50.394918+0000","last_clean":"2024-08-31T22:12:50.394918+0000","last_became_active":"2024-08-31T22:12:48.390967+0000","last_became_peered":"2024-08-31T22:12:48.390967+0000","last_unstale":"2024-08-31T22:12:50.394918+0000","last_undegraded":"2024-08-31T22:12:50.394918+0000","last_fullsized":"2024-08-31T22:12:50.394918+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T07:18:29.001799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002301787,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:12:50.393340+0000","last_change":"2024-08-31T22:12:50.393340+0000","last_active":"2024-08-31T22:12:50.393340+0000","last_peered":"2024-08-31T22:12:50.393340+0000","last_clean":"2024-08-31T22:12:50.393340+0000","last_became_active":"2024-08-31T22:12:48.393810+0000","last_became_peered":"2024-08-31T22:12:48.393810+0000","last_unstale":"2024-08-31T22:12:50.393340+0000","last_undegraded":"2024-08-31T22:12:50.393340+0000","last_fullsized":"2024-08-31T22:12:50.393340+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:37:46.050434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011422590000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:12:50.394198+0000","last_change":"2024-08-31T22:12:50.394198+0000","last_active":"2024-08-31T22:12:50.394198+0000","last_peered":"2024-08-31T22:12:50.394198+0000","last_clean":"2024-08-31T22:12:50.394198+0000","last_became_active":"2024-08-31T22:12:48.395901+0000","last_became_peered":"2024-08-31T22:12:48.395901+0000","last_unstale":"2024-08-31T22:12:50.394198+0000","last_undegraded":"2024-08-31T22:12:50.394198+0000","last_fullsized":"2024-08-31T22:12:50.394198+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T02:29:23.840873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018848599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"17'2","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:12:50.403426+0000","last_change":"2024-08-31T22:12:50.400538+0000","last_active":"2024-08-31T22:12:50.403426+0000","last_peered":"2024-08-31T22:12:50.403426+0000","last_clean":"2024-08-31T22:12:50.403426+0000","last_became_active":"2024-08-31T22:12:48.389795+0000","last_became_peered":"2024-08-31T22:12:48.389795+0000","last_unstale":"2024-08-31T22:12:50.403426+0000","last_undegraded":"2024-08-31T22:12:50.403426+0000","last_fullsized":"2024-08-31T22:12:50.403426+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T01:35:29.294687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023100410000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:12:51.173642+0000","last_change":"2024-08-31T22:12:51.173642+0000","last_active":"2024-08-31T22:12:51.173642+0000","last_peered":"2024-08-31T22:12:51.173642+0000","last_clean":"2024-08-31T22:12:51.173642+0000","last_became_active":"2024-08-31T22:12:48.391597+0000","last_became_peered":"2024-08-31T22:12:48.391597+0000","last_unstale":"2024-08-31T22:12:51.173642+0000","last_undegraded":"2024-08-31T22:12:51.173642+0000","last_fullsized":"2024-08-31T22:12:51.173642+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:22:09.506410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051578099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:12:51.173571+0000","last_change":"2024-08-31T22:12:51.173571+0000","last_active":"2024-08-31T22:12:51.173571+0000","last_peered":"2024-08-31T22:12:51.173571+0000","last_clean":"2024-08-31T22:12:51.173571+0000","last_became_active":"2024-08-31T22:12:48.393856+0000","last_became_peered":"2024-08-31T22:12:48.393856+0000","last_unstale":"2024-08-31T22:12:51.173571+0000","last_undegraded":"2024-08-31T22:12:51.173571+0000","last_fullsized":"2024-08-31T22:12:51.173571+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:15:04.841041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046582699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:12:50.393581+0000","last_change":"2024-08-31T22:12:50.393581+0000","last_active":"2024-08-31T22:12:50.393581+0000","last_peered":"2024-08-31T22:12:50.393581+0000","last_clean":"2024-08-31T22:12:50.393581+0000","last_became_active":"2024-08-31T22:12:48.393780+0000","last_became_peered":"2024-08-31T22:12:48.393780+0000","last_unstale":"2024-08-31T22:12:50.393581+0000","last_undegraded":"2024-08-31T22:12:50.393581+0000","last_fullsized":"2024-08-31T22:12:50.393581+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T01:27:52.811385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013191050000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'97","reported_seq":147,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:12:50.391952+0000","last_change":"2024-08-31T22:12:44.358595+0000","last_active":"2024-08-31T22:12:50.391952+0000","last_peered":"2024-08-31T22:12:50.391952+0000","last_clean":"2024-08-31T22:12:50.391952+0000","last_became_active":"2024-08-31T22:12:44.357729+0000","last_became_peered":"2024-08-31T22:12:44.357729+0000","last_unstale":"2024-08-31T22:12:50.391952+0000","last_undegraded":"2024-08-31T22:12:50.391952+0000","last_fullsized":"2024-08-31T22:12:50.391952+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:43.340061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:43.340061+0000","last_clean_scrub_stamp":"2024-08-31T22:12:43.340061+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:14:47.042415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6360,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31324,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340516,"statfs":{"total":96636764160,"available":96604688384,"internally_reserved":0,"allocated":139264,"data_stored":39652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":208896,"data_stored":633601,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":208896,"data_stored":633601,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T22:12:53.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-08-31T22:12:53.986 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:53.987 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-08-31T22:12:54.003 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-08-31T22:12:52.313280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":94120,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283021400,"statfs":{"total":289910292480,"available":289813913600,"internally_reserved":0,"allocated":557056,"data_stored":1306854,"data_compressed":6360,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.939375"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:12:50.394482+0000","last_change":"2024-08-31T22:12:50.394482+0000","last_active":"2024-08-31T22:12:50.394482+0000","last_peered":"2024-08-31T22:12:50.394482+0000","last_clean":"2024-08-31T22:12:50.394482+0000","last_became_active":"2024-08-31T22:12:48.393530+0000","last_became_peered":"2024-08-31T22:12:48.393530+0000","last_unstale":"2024-08-31T22:12:50.394482+0000","last_undegraded":"2024-08-31T22:12:50.394482+0000","last_fullsized":"2024-08-31T22:12:50.394482+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:53:59.887015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021475320000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:12:50.394918+0000","last_change":"2024-08-31T22:12:50.394918+0000","last_active":"2024-08-31T22:12:50.394918+0000","last_peered":"2024-08-31T22:12:50.394918+0000","last_clean":"2024-08-31T22:12:50.394918+0000","last_became_active":"2024-08-31T22:12:48.390967+0000","last_became_peered":"2024-08-31T22:12:48.390967+0000","last_unstale":"2024-08-31T22:12:50.394918+0000","last_undegraded":"2024-08-31T22:12:50.394918+0000","last_fullsized":"2024-08-31T22:12:50.394918+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T07:18:29.001799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002301787,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:12:50.393340+0000","last_change":"2024-08-31T22:12:50.393340+0000","last_active":"2024-08-31T22:12:50.393340+0000","last_peered":"2024-08-31T22:12:50.393340+0000","last_clean":"2024-08-31T22:12:50.393340+0000","last_became_active":"2024-08-31T22:12:48.393810+0000","last_became_peered":"2024-08-31T22:12:48.393810+0000","last_unstale":"2024-08-31T22:12:50.393340+0000","last_undegraded":"2024-08-31T22:12:50.393340+0000","last_fullsized":"2024-08-31T22:12:50.393340+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:37:46.050434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011422590000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:12:50.394198+0000","last_change":"2024-08-31T22:12:50.394198+0000","last_active":"2024-08-31T22:12:50.394198+0000","last_peered":"2024-08-31T22:12:50.394198+0000","last_clean":"2024-08-31T22:12:50.394198+0000","last_became_active":"2024-08-31T22:12:48.395901+0000","last_became_peered":"2024-08-31T22:12:48.395901+0000","last_unstale":"2024-08-31T22:12:50.394198+0000","last_undegraded":"2024-08-31T22:12:50.394198+0000","last_fullsized":"2024-08-31T22:12:50.394198+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T02:29:23.840873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018848599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"17'2","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:12:50.403426+0000","last_change":"2024-08-31T22:12:50.400538+0000","last_active":"2024-08-31T22:12:50.403426+0000","last_peered":"2024-08-31T22:12:50.403426+0000","last_clean":"2024-08-31T22:12:50.403426+0000","last_became_active":"2024-08-31T22:12:48.389795+0000","last_became_peered":"2024-08-31T22:12:48.389795+0000","last_unstale":"2024-08-31T22:12:50.403426+0000","last_undegraded":"2024-08-31T22:12:50.403426+0000","last_fullsized":"2024-08-31T22:12:50.403426+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T01:35:29.294687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023100410000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:12:51.173642+0000","last_change":"2024-08-31T22:12:51.173642+0000","last_active":"2024-08-31T22:12:51.173642+0000","last_peered":"2024-08-31T22:12:51.173642+0000","last_clean":"2024-08-31T22:12:51.173642+0000","last_became_active":"2024-08-31T22:12:48.391597+0000","last_became_peered":"2024-08-31T22:12:48.391597+0000","last_unstale":"2024-08-31T22:12:51.173642+0000","last_undegraded":"2024-08-31T22:12:51.173642+0000","last_fullsized":"2024-08-31T22:12:51.173642+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:22:09.506410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051578099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:12:51.173571+0000","last_change":"2024-08-31T22:12:51.173571+0000","last_active":"2024-08-31T22:12:51.173571+0000","last_peered":"2024-08-31T22:12:51.173571+0000","last_clean":"2024-08-31T22:12:51.173571+0000","last_became_active":"2024-08-31T22:12:48.393856+0000","last_became_peered":"2024-08-31T22:12:48.393856+0000","last_unstale":"2024-08-31T22:12:51.173571+0000","last_undegraded":"2024-08-31T22:12:51.173571+0000","last_fullsized":"2024-08-31T22:12:51.173571+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:15:04.841041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046582699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:12:50.393581+0000","last_change":"2024-08-31T22:12:50.393581+0000","last_active":"2024-08-31T22:12:50.393581+0000","last_peered":"2024-08-31T22:12:50.393581+0000","last_clean":"2024-08-31T22:12:50.393581+0000","last_became_active":"2024-08-31T22:12:48.393780+0000","last_became_peered":"2024-08-31T22:12:48.393780+0000","last_unstale":"2024-08-31T22:12:50.393581+0000","last_undegraded":"2024-08-31T22:12:50.393581+0000","last_fullsized":"2024-08-31T22:12:50.393581+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T01:27:52.811385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013191050000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'97","reported_seq":147,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:12:50.391952+0000","last_change":"2024-08-31T22:12:44.358595+0000","last_active":"2024-08-31T22:12:50.391952+0000","last_peered":"2024-08-31T22:12:50.391952+0000","last_clean":"2024-08-31T22:12:50.391952+0000","last_became_active":"2024-08-31T22:12:44.357729+0000","last_became_peered":"2024-08-31T22:12:44.357729+0000","last_unstale":"2024-08-31T22:12:50.391952+0000","last_undegraded":"2024-08-31T22:12:50.391952+0000","last_fullsized":"2024-08-31T22:12:50.391952+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:43.340061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:43.340061+0000","last_clean_scrub_stamp":"2024-08-31T22:12:43.340061+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:14:47.042415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6360,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31324,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340516,"statfs":{"total":96636764160,"available":96604688384,"internally_reserved":0,"allocated":139264,"data_stored":39652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":208896,"data_stored":633601,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":208896,"data_stored":633601,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T22:12:54.004 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-08-31T22:12:54.004 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-08-31T22:12:54.004 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-08-31T22:12:54.004 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-08-31T22:12:54.319 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:54.320 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-31T22:12:54.337 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-08-31T22:12:54.337 INFO:teuthology.run_tasks:Running task ceph... 2024-08-31T22:12:54.347 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-08-31T22:12:54.347 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-08-31T22:12:54.347 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-08-31_21:16:02-rbd-squid-distro-default-smithi/7883608', 'branch': 'squid', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7883608', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-08-31_21:16:02-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '3091dfc0187f68df2afd4e392dc4c381cfdc1cad'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3091dfc0187f68df2afd4e392dc4c381cfdc1cad'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': '3091dfc0187f68df2afd4e392dc4c381cfdc1cad'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 552, 'sha1': '3091dfc0187f68df2afd4e392dc4c381cfdc1cad', 'sleep_before_teardown': 0, 'subset': '40/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_3091dfc0187f68df2afd4e392dc4c381cfdc1cad/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '3091dfc0187f68df2afd4e392dc4c381cfdc1cad', 'targets': {'smithi007.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPXUeOWcSMUCUypX4YCMcgYYBi9GKauqGGUmH0fi5vzIeU6SJbiht+tepppB/MtJQna3sVZ2fKQUW0qywuwh218=', 'smithi032.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC9ImDX2yUmt3UPP3yhZ5c5Z5ePr116mvTICgiHMVQJB4QGQf95OWkNXcWQYcyswUbvK1EMz1+9d/gLndRc9F3M='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '3091dfc0187f68df2afd4e392dc4c381cfdc1cad'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '3091dfc0187f68df2afd4e392dc4c381cfdc1cad'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '3091dfc0187f68df2afd4e392dc4c381cfdc1cad'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_MIN_COMPAT_CLIENT': 'octopus', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-08-31_21:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-08-31T22:12:54.348 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-08-31T22:12:54.350 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-08-31T22:12:54.356 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/run/ceph 2024-08-31T22:12:54.398 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m0777 -- /var/run/ceph 2024-08-31T22:12:54.413 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:54.414 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2024-08-31T22:12:54.461 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-31T22:12:54.461 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_1 2024-08-31T22:12:54.509 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-31T22:12:54.510 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-31T22:12:54.510 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 878 Links: 1 2024-08-31T22:12:54.510 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-31T22:12:54.510 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-08-31 22:06:41.056210628 +0000 2024-08-31T22:12:54.510 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-08-31 22:06:40.812212805 +0000 2024-08-31T22:12:54.510 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-08-31 22:06:40.812212805 +0000 2024-08-31T22:12:54.510 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-08-31T22:12:54.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-31T22:12:54.563 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-08-31T22:12:54.564 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-08-31T22:12:54.564 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000303995 s, 1.7 MB/s 2024-08-31T22:12:54.565 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-31T22:12:54.614 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_2 2024-08-31T22:12:54.665 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-31T22:12:54.665 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-31T22:12:54.665 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 890 Links: 1 2024-08-31T22:12:54.665 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-31T22:12:54.665 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-08-31 22:06:41.536206345 +0000 2024-08-31T22:12:54.665 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-08-31 22:06:41.296208487 +0000 2024-08-31T22:12:54.665 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-08-31 22:06:41.296208487 +0000 2024-08-31T22:12:54.665 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-08-31T22:12:54.666 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-31T22:12:54.719 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-08-31T22:12:54.719 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-08-31T22:12:54.719 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000305595 s, 1.7 MB/s 2024-08-31T22:12:54.721 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-31T22:12:54.770 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_3 2024-08-31T22:12:54.817 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-31T22:12:54.818 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-31T22:12:54.818 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 913 Links: 1 2024-08-31T22:12:54.818 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-31T22:12:54.818 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-08-31 22:06:42.036201886 +0000 2024-08-31T22:12:54.818 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-08-31 22:06:41.780204169 +0000 2024-08-31T22:12:54.818 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-08-31 22:06:41.780204169 +0000 2024-08-31T22:12:54.818 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-08-31T22:12:54.818 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-31T22:12:54.872 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-08-31T22:12:54.872 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-08-31T22:12:54.873 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000318533 s, 1.6 MB/s 2024-08-31T22:12:54.874 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-31T22:12:54.922 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_4 2024-08-31T22:12:54.969 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-31T22:12:54.970 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-31T22:12:54.970 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-08-31T22:12:54.970 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-31T22:12:54.970 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-08-31 22:06:42.504197711 +0000 2024-08-31T22:12:54.970 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-08-31 22:06:42.256199923 +0000 2024-08-31T22:12:54.970 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-08-31 22:06:42.256199923 +0000 2024-08-31T22:12:54.970 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-08-31T22:12:54.970 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-31T22:12:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-08-31T22:12:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-08-31T22:12:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000298637 s, 1.7 MB/s 2024-08-31T22:12:55.025 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-31T22:12:55.074 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-31T22:12:55.074 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi032.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-08-31T22:12:55.074 INFO:tasks.ceph:Generating config... 2024-08-31T22:12:55.075 INFO:tasks.ceph:[client] debug rbd = 20 2024-08-31T22:12:55.075 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-08-31T22:12:55.076 INFO:tasks.ceph:[client] log to stderr = False 2024-08-31T22:12:55.076 INFO:tasks.ceph:[client] rbd default features = 125 2024-08-31T22:12:55.076 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.076 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.076 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.076 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.076 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.076 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.076 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.076 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.076 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.076 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.076 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.076 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.077 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.077 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.077 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-08-31T22:12:55.077 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-08-31T22:12:55.077 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-08-31T22:12:55.077 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-08-31T22:12:55.077 INFO:tasks.ceph:[mgr] debug ms = 1 2024-08-31T22:12:55.077 INFO:tasks.ceph:[mon] debug mon = 20 2024-08-31T22:12:55.077 INFO:tasks.ceph:[mon] debug ms = 1 2024-08-31T22:12:55.077 INFO:tasks.ceph:[mon] debug paxos = 20 2024-08-31T22:12:55.077 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-08-31T22:12:55.077 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-08-31T22:12:55.077 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-08-31T22:12:55.077 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-08-31T22:12:55.078 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-08-31T22:12:55.078 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-08-31T22:12:55.078 INFO:tasks.ceph:[osd] debug ms = 1 2024-08-31T22:12:55.078 INFO:tasks.ceph:[osd] debug osd = 20 2024-08-31T22:12:55.078 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-08-31T22:12:55.078 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-08-31T22:12:55.078 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-08-31T22:12:55.078 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-08-31T22:12:55.078 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-08-31T22:12:55.078 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-08-31T22:12:55.078 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-08-31T22:12:55.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-08-31T22:12:55.149 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/cluster2.keyring 2024-08-31T22:12:55.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-08-31T22:12:55.232 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-08-31T22:12:55.293 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.32')] 2024-08-31T22:12:55.293 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.32', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-08-31T22:12:55.294 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:55.294 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-08-31T22:12:55.342 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.32 --print /home/ubuntu/cephtest/cluster2.monmap 2024-08-31T22:12:55.422 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-08-31T22:12:55.422 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: generated fsid 2454aa5a-0087-4f83-8364-979b8131b8af 2024-08-31T22:12:55.422 INFO:teuthology.orchestra.run.smithi032.stdout:setting min_mon_release = quincy 2024-08-31T22:12:55.422 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 0 2024-08-31T22:12:55.422 INFO:teuthology.orchestra.run.smithi032.stdout:fsid 2454aa5a-0087-4f83-8364-979b8131b8af 2024-08-31T22:12:55.422 INFO:teuthology.orchestra.run.smithi032.stdout:last_changed 2024-08-31T22:12:55.424338+0000 2024-08-31T22:12:55.422 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-08-31T22:12:55.424338+0000 2024-08-31T22:12:55.422 INFO:teuthology.orchestra.run.smithi032.stdout:min_mon_release 17 (quincy) 2024-08-31T22:12:55.423 INFO:teuthology.orchestra.run.smithi032.stdout:election_strategy: 1 2024-08-31T22:12:55.423 INFO:teuthology.orchestra.run.smithi032.stdout:0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-08-31T22:12:55.423 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-08-31T22:12:55.424 DEBUG:teuthology.orchestra.run.smithi032:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-08-31T22:12:55.473 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 2454aa5a-0087-4f83-8364-979b8131b8af... 2024-08-31T22:12:55.474 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-08-31T22:12:55.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-08-31T22:12:55.519 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2024-08-31T22:12:55.520 INFO:teuthology.orchestra.run.smithi007.stdout: chdir = "" 2024-08-31T22:12:55.520 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-31T22:12:55.520 INFO:teuthology.orchestra.run.smithi007.stdout: auth supported = cephx 2024-08-31T22:12:55.520 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.520 INFO:teuthology.orchestra.run.smithi007.stdout: filestore xattr use omap = true 2024-08-31T22:12:55.520 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.520 INFO:teuthology.orchestra.run.smithi007.stdout: mon clock drift allowed = 1.000 2024-08-31T22:12:55.520 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.520 INFO:teuthology.orchestra.run.smithi007.stdout: osd crush chooseleaf type = 0 2024-08-31T22:12:55.520 INFO:teuthology.orchestra.run.smithi007.stdout: auth debug = true 2024-08-31T22:12:55.520 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.521 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on old message = true 2024-08-31T22:12:55.521 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on bug = true 2024-08-31T22:12:55.521 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.521 INFO:teuthology.orchestra.run.smithi007.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-31T22:12:55.521 INFO:teuthology.orchestra.run.smithi007.stdout: mon pg warn max object skew = 0 2024-08-31T22:12:55.521 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.521 INFO:teuthology.orchestra.run.smithi007.stdout: # disable pg_autoscaler by default for new pools 2024-08-31T22:12:55.521 INFO:teuthology.orchestra.run.smithi007.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-31T22:12:55.521 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.521 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default size = 2 2024-08-31T22:12:55.521 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.522 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow primary affinity = true 2024-08-31T22:12:55.522 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow pg remap = true 2024-08-31T22:12:55.522 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on legacy crush tunables = false 2024-08-31T22:12:55.522 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on crush straw calc version zero = false 2024-08-31T22:12:55.522 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on no sortbitwise = false 2024-08-31T22:12:55.522 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on osd down out interval zero = false 2024-08-31T22:12:55.522 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on too few osds = false 2024-08-31T22:12:55.523 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-31T22:12:55.523 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-31T22:12:55.523 INFO:teuthology.orchestra.run.smithi007.stdout: mon_allow_pool_size_one = true 2024-08-31T22:12:55.523 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.523 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-31T22:12:55.523 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.523 INFO:teuthology.orchestra.run.smithi007.stdout: osd default data pool replay window = 5 2024-08-31T22:12:55.523 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.523 INFO:teuthology.orchestra.run.smithi007.stdout: mon allow pool delete = true 2024-08-31T22:12:55.524 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.524 INFO:teuthology.orchestra.run.smithi007.stdout: mon cluster log file level = debug 2024-08-31T22:12:55.524 INFO:teuthology.orchestra.run.smithi007.stdout: debug asserts on shutdown = true 2024-08-31T22:12:55.524 INFO:teuthology.orchestra.run.smithi007.stdout: mon health detail to clog = false 2024-08-31T22:12:55.524 INFO:teuthology.orchestra.run.smithi007.stdout: mon host = 172.21.15.32 2024-08-31T22:12:55.524 INFO:teuthology.orchestra.run.smithi007.stdout: mon client directed command retry = 5 2024-08-31T22:12:55.524 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on pool no app = False 2024-08-31T22:12:55.524 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject socket failures = 5000 2024-08-31T22:12:55.524 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = 2454aa5a-0087-4f83-8364-979b8131b8af 2024-08-31T22:12:55.524 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.525 INFO:teuthology.orchestra.run.smithi007.stdout:[osd] 2024-08-31T22:12:55.525 INFO:teuthology.orchestra.run.smithi007.stdout: osd journal size = 100 2024-08-31T22:12:55.525 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.525 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub load threshold = 5.0 2024-08-31T22:12:55.525 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub max interval = 600 2024-08-31T22:12:55.525 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock profile = high_recovery_ops 2024-08-31T22:12:55.525 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.525 INFO:teuthology.orchestra.run.smithi007.stdout: osd recover clone overlap = true 2024-08-31T22:12:55.525 INFO:teuthology.orchestra.run.smithi007.stdout: osd recovery max chunk = 1048576 2024-08-31T22:12:55.525 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.525 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug shutdown = true 2024-08-31T22:12:55.525 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug op order = true 2024-08-31T22:12:55.525 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify stray on activate = true 2024-08-31T22:12:55.525 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug trim objects = true 2024-08-31T22:12:55.526 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.526 INFO:teuthology.orchestra.run.smithi007.stdout: osd open classes on start = true 2024-08-31T22:12:55.526 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug pg log writeout = true 2024-08-31T22:12:55.526 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.526 INFO:teuthology.orchestra.run.smithi007.stdout: osd deep scrub update digest min age = 30 2024-08-31T22:12:55.526 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.526 INFO:teuthology.orchestra.run.smithi007.stdout: osd map max advance = 10 2024-08-31T22:12:55.526 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.526 INFO:teuthology.orchestra.run.smithi007.stdout: journal zero on create = true 2024-08-31T22:12:55.526 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.526 INFO:teuthology.orchestra.run.smithi007.stdout: filestore ondisk finisher threads = 3 2024-08-31T22:12:55.526 INFO:teuthology.orchestra.run.smithi007.stdout: filestore apply finisher threads = 3 2024-08-31T22:12:55.526 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.526 INFO:teuthology.orchestra.run.smithi007.stdout: bdev debug aio = true 2024-08-31T22:12:55.527 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug misdirected ops = true 2024-08-31T22:12:55.527 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore block size = 96636764160 2024-08-31T22:12:55.527 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore compression algorithm = zstd 2024-08-31T22:12:55.527 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore compression mode = aggressive 2024-08-31T22:12:55.527 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore fsck on mount = True 2024-08-31T22:12:55.527 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluefs = 1/20 2024-08-31T22:12:55.527 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluestore = 1/20 2024-08-31T22:12:55.527 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-08-31T22:12:55.527 INFO:teuthology.orchestra.run.smithi007.stdout: debug osd = 20 2024-08-31T22:12:55.527 INFO:teuthology.orchestra.run.smithi007.stdout: debug rocksdb = 4/10 2024-08-31T22:12:55.527 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd backfillfull_ratio = 0.85 2024-08-31T22:12:55.527 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd full ratio = 0.9 2024-08-31T22:12:55.527 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd nearfull ratio = 0.8 2024-08-31T22:12:55.528 INFO:teuthology.orchestra.run.smithi007.stdout: osd failsafe full ratio = 0.95 2024-08-31T22:12:55.528 INFO:teuthology.orchestra.run.smithi007.stdout: osd objectstore = bluestore 2024-08-31T22:12:55.528 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.528 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr] 2024-08-31T22:12:55.528 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-08-31T22:12:55.528 INFO:teuthology.orchestra.run.smithi007.stdout: debug mgr = 20 2024-08-31T22:12:55.528 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2024-08-31T22:12:55.528 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-08-31T22:12:55.528 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-08-31T22:12:55.528 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-08-31T22:12:55.528 INFO:teuthology.orchestra.run.smithi007.stdout: mgr/telemetry/nag = false 2024-08-31T22:12:55.528 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.528 INFO:teuthology.orchestra.run.smithi007.stdout:[mon] 2024-08-31T22:12:55.528 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-08-31T22:12:55.529 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2024-08-31T22:12:55.529 INFO:teuthology.orchestra.run.smithi007.stdout: debug paxos = 20 2024-08-31T22:12:55.529 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-08-31T22:12:55.529 INFO:teuthology.orchestra.run.smithi007.stdout: mon data avail warn = 5 2024-08-31T22:12:55.529 INFO:teuthology.orchestra.run.smithi007.stdout: mon mgr mkfs grace = 240 2024-08-31T22:12:55.529 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-08-31T22:12:55.529 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd reporter subtree level = osd 2024-08-31T22:12:55.529 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd prime pg temp = true 2024-08-31T22:12:55.529 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-08-31T22:12:55.529 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.529 INFO:teuthology.orchestra.run.smithi007.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-31T22:12:55.529 INFO:teuthology.orchestra.run.smithi007.stdout: auth mon ticket ttl = 660 # 11m 2024-08-31T22:12:55.529 INFO:teuthology.orchestra.run.smithi007.stdout: auth service ticket ttl = 240 # 4m 2024-08-31T22:12:55.529 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.530 INFO:teuthology.orchestra.run.smithi007.stdout: # don't complain about insecure global_id in the test suite 2024-08-31T22:12:55.530 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-31T22:12:55.530 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-31T22:12:55.530 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.530 INFO:teuthology.orchestra.run.smithi007.stdout: # 1m isn't quite enough 2024-08-31T22:12:55.530 INFO:teuthology.orchestra.run.smithi007.stdout: mon_down_mkfs_grace = 2m 2024-08-31T22:12:55.530 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.530 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_filestore_osds = false 2024-08-31T22:12:55.530 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:12:55.530 INFO:teuthology.orchestra.run.smithi007.stdout:[client] 2024-08-31T22:12:55.530 INFO:teuthology.orchestra.run.smithi007.stdout: rgw cache enabled = true 2024-08-31T22:12:55.530 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable ops log = true 2024-08-31T22:12:55.530 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable usage log = true 2024-08-31T22:12:55.531 INFO:teuthology.orchestra.run.smithi007.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-31T22:12:55.531 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-31T22:12:55.531 INFO:teuthology.orchestra.run.smithi007.stdout: debug rbd = 20 2024-08-31T22:12:55.531 INFO:teuthology.orchestra.run.smithi007.stdout: debug rbd_mirror = 15 2024-08-31T22:12:55.531 INFO:teuthology.orchestra.run.smithi007.stdout: log to stderr = False 2024-08-31T22:12:55.531 INFO:teuthology.orchestra.run.smithi007.stdout: rbd default features = 125 2024-08-31T22:12:55.531 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.a] 2024-08-31T22:12:55.531 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.0] 2024-08-31T22:12:55.531 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.531 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.531 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.1] 2024-08-31T22:12:55.531 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.531 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.531 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.2] 2024-08-31T22:12:55.532 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.532 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.532 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.3] 2024-08-31T22:12:55.532 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.532 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.532 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.4] 2024-08-31T22:12:55.532 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.532 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.532 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.5] 2024-08-31T22:12:55.532 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.532 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.532 INFO:teuthology.orchestra.run.smithi007.stdout:[client.mirror.6] 2024-08-31T22:12:55.532 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.532 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.547 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-08-31T22:12:55.547 INFO:teuthology.orchestra.run.smithi032.stdout: chdir = "" 2024-08-31T22:12:55.547 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-31T22:12:55.547 INFO:teuthology.orchestra.run.smithi032.stdout: auth supported = cephx 2024-08-31T22:12:55.547 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.547 INFO:teuthology.orchestra.run.smithi032.stdout: filestore xattr use omap = true 2024-08-31T22:12:55.548 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.548 INFO:teuthology.orchestra.run.smithi032.stdout: mon clock drift allowed = 1.000 2024-08-31T22:12:55.548 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.548 INFO:teuthology.orchestra.run.smithi032.stdout: osd crush chooseleaf type = 0 2024-08-31T22:12:55.548 INFO:teuthology.orchestra.run.smithi032.stdout: auth debug = true 2024-08-31T22:12:55.548 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.548 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on old message = true 2024-08-31T22:12:55.548 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on bug = true 2024-08-31T22:12:55.548 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.548 INFO:teuthology.orchestra.run.smithi032.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-31T22:12:55.548 INFO:teuthology.orchestra.run.smithi032.stdout: mon pg warn max object skew = 0 2024-08-31T22:12:55.548 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.548 INFO:teuthology.orchestra.run.smithi032.stdout: # disable pg_autoscaler by default for new pools 2024-08-31T22:12:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-31T22:12:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default size = 2 2024-08-31T22:12:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow primary affinity = true 2024-08-31T22:12:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow pg remap = true 2024-08-31T22:12:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on legacy crush tunables = false 2024-08-31T22:12:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on crush straw calc version zero = false 2024-08-31T22:12:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on no sortbitwise = false 2024-08-31T22:12:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on osd down out interval zero = false 2024-08-31T22:12:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on too few osds = false 2024-08-31T22:12:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-31T22:12:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-31T22:12:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: mon_allow_pool_size_one = true 2024-08-31T22:12:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-31T22:12:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: osd default data pool replay window = 5 2024-08-31T22:12:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: mon allow pool delete = true 2024-08-31T22:12:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: mon cluster log file level = debug 2024-08-31T22:12:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: debug asserts on shutdown = true 2024-08-31T22:12:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: mon health detail to clog = false 2024-08-31T22:12:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: mon host = 172.21.15.32 2024-08-31T22:12:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: mon client directed command retry = 5 2024-08-31T22:12:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on pool no app = False 2024-08-31T22:12:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: ms inject socket failures = 5000 2024-08-31T22:12:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = 2454aa5a-0087-4f83-8364-979b8131b8af 2024-08-31T22:12:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.551 INFO:teuthology.orchestra.run.smithi032.stdout:[osd] 2024-08-31T22:12:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: osd journal size = 100 2024-08-31T22:12:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub load threshold = 5.0 2024-08-31T22:12:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub max interval = 600 2024-08-31T22:12:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: osd mclock profile = high_recovery_ops 2024-08-31T22:12:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: osd recover clone overlap = true 2024-08-31T22:12:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: osd recovery max chunk = 1048576 2024-08-31T22:12:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug shutdown = true 2024-08-31T22:12:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug op order = true 2024-08-31T22:12:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify stray on activate = true 2024-08-31T22:12:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug trim objects = true 2024-08-31T22:12:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: osd open classes on start = true 2024-08-31T22:12:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug pg log writeout = true 2024-08-31T22:12:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: osd deep scrub update digest min age = 30 2024-08-31T22:12:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: osd map max advance = 10 2024-08-31T22:12:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: journal zero on create = true 2024-08-31T22:12:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: filestore ondisk finisher threads = 3 2024-08-31T22:12:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: filestore apply finisher threads = 3 2024-08-31T22:12:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: bdev debug aio = true 2024-08-31T22:12:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug misdirected ops = true 2024-08-31T22:12:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore block size = 96636764160 2024-08-31T22:12:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore compression algorithm = zstd 2024-08-31T22:12:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore compression mode = aggressive 2024-08-31T22:12:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore fsck on mount = True 2024-08-31T22:12:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluefs = 1/20 2024-08-31T22:12:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluestore = 1/20 2024-08-31T22:12:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-08-31T22:12:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: debug osd = 20 2024-08-31T22:12:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: debug rocksdb = 4/10 2024-08-31T22:12:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd backfillfull_ratio = 0.85 2024-08-31T22:12:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd full ratio = 0.9 2024-08-31T22:12:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd nearfull ratio = 0.8 2024-08-31T22:12:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: osd failsafe full ratio = 0.95 2024-08-31T22:12:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: osd objectstore = bluestore 2024-08-31T22:12:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.554 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr] 2024-08-31T22:12:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-08-31T22:12:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: debug mgr = 20 2024-08-31T22:12:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-08-31T22:12:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-08-31T22:12:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-08-31T22:12:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-08-31T22:12:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: mgr/telemetry/nag = false 2024-08-31T22:12:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.555 INFO:teuthology.orchestra.run.smithi032.stdout:[mon] 2024-08-31T22:12:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-08-31T22:12:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-08-31T22:12:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: debug paxos = 20 2024-08-31T22:12:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-08-31T22:12:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: mon data avail warn = 5 2024-08-31T22:12:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: mon mgr mkfs grace = 240 2024-08-31T22:12:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-08-31T22:12:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd reporter subtree level = osd 2024-08-31T22:12:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd prime pg temp = true 2024-08-31T22:12:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-08-31T22:12:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-31T22:12:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: auth mon ticket ttl = 660 # 11m 2024-08-31T22:12:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: auth service ticket ttl = 240 # 4m 2024-08-31T22:12:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: # don't complain about insecure global_id in the test suite 2024-08-31T22:12:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-31T22:12:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-31T22:12:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: # 1m isn't quite enough 2024-08-31T22:12:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: mon_down_mkfs_grace = 2m 2024-08-31T22:12:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_filestore_osds = false 2024-08-31T22:12:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:12:55.557 INFO:teuthology.orchestra.run.smithi032.stdout:[client] 2024-08-31T22:12:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: rgw cache enabled = true 2024-08-31T22:12:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable ops log = true 2024-08-31T22:12:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable usage log = true 2024-08-31T22:12:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-31T22:12:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-31T22:12:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: debug rbd = 20 2024-08-31T22:12:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: debug rbd_mirror = 15 2024-08-31T22:12:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: log to stderr = False 2024-08-31T22:12:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: rbd default features = 125 2024-08-31T22:12:55.557 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.a] 2024-08-31T22:12:55.557 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.0] 2024-08-31T22:12:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.558 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.1] 2024-08-31T22:12:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.558 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.2] 2024-08-31T22:12:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.558 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.3] 2024-08-31T22:12:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.558 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.4] 2024-08-31T22:12:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.559 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.5] 2024-08-31T22:12:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.559 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.6] 2024-08-31T22:12:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-08-31T22:12:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-08-31T22:12:55.560 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-08-31T22:12:55.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-08-31T22:12:55.653 INFO:tasks.ceph:Copying monmap to all nodes... 2024-08-31T22:12:55.653 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:55.653 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-08-31T22:12:55.658 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:55.658 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-08-31T22:12:55.705 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi007.front.sepia.ceph.com 2024-08-31T22:12:55.705 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:55.705 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/cluster2.keyring 2024-08-31T22:12:55.705 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-08-31T22:12:55.729 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:12:55.729 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-08-31T22:12:55.777 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi032.front.sepia.ceph.com 2024-08-31T22:12:55.778 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:55.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/cluster2.keyring 2024-08-31T22:12:55.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-08-31T22:12:55.799 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:12:55.799 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-08-31T22:12:55.845 INFO:tasks.ceph:Setting up mon nodes... 2024-08-31T22:12:55.845 INFO:tasks.ceph:Setting up mgr nodes... 2024-08-31T22:12:55.845 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-08-31T22:12:55.893 INFO:teuthology.orchestra.run.smithi007.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-08-31T22:12:55.896 INFO:tasks.ceph:Setting up mds nodes... 2024-08-31T22:12:55.896 INFO:tasks.ceph_client:Setting up client nodes... 2024-08-31T22:12:55.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-08-31T22:12:55.928 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-08-31T22:12:55.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-08-31T22:12:55.976 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-08-31T22:12:55.987 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-08-31T22:12:56.065 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-08-31T22:12:56.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-08-31T22:12:56.152 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-08-31T22:12:56.163 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-08-31T22:12:56.239 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-08-31T22:12:56.251 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-08-31T22:12:56.327 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-08-31T22:12:56.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-08-31T22:12:56.412 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-08-31T22:12:56.423 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-08-31T22:12:56.499 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-08-31T22:12:56.509 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-08-31T22:12:56.583 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-08-31T22:12:56.593 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-08-31T22:12:56.593 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi032.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-08-31T22:12:56.594 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-08-31T22:12:56.648 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-31T22:12:56.648 INFO:tasks.ceph:role: cluster2.osd.0 2024-08-31T22:12:56.648 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi032.front.sepia.ceph.com 2024-08-31T22:12:56.648 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-31T22:12:56.704 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-31T22:12:56.704 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-31T22:12:56.704 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-31T22:12:56.704 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-08-31T22:12:56.704 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-31T22:12:56.704 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-08-31T22:12:56.705 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-31T22:12:56.705 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-08-31T22:12:56.705 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-31T22:12:56.705 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-31T22:12:56.729 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-08-31T22:12:56.730 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-08-31T22:12:56.731 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-08-31T22:12:56.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-08-31T22:12:56.845 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: /sbin/restorecon: command not found 2024-08-31T22:12:56.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T22:12:56.846 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-08-31T22:12:56.898 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-31T22:12:56.899 INFO:tasks.ceph:role: cluster2.osd.1 2024-08-31T22:12:56.899 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi032.front.sepia.ceph.com 2024-08-31T22:12:56.899 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-31T22:12:56.953 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-31T22:12:56.953 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-31T22:12:56.953 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-31T22:12:56.953 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-08-31T22:12:56.953 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-31T22:12:56.954 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-08-31T22:12:56.954 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-31T22:12:56.954 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-08-31T22:12:56.954 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-31T22:12:56.954 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-31T22:12:56.976 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-08-31T22:12:56.978 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-08-31T22:12:56.978 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-08-31T22:12:57.036 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-08-31T22:12:57.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T22:12:57.090 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: /sbin/restorecon: command not found 2024-08-31T22:12:57.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-08-31T22:12:57.144 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-31T22:12:57.144 INFO:tasks.ceph:role: cluster2.osd.2 2024-08-31T22:12:57.144 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi032.front.sepia.ceph.com 2024-08-31T22:12:57.144 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-31T22:12:57.200 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-31T22:12:57.201 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-31T22:12:57.201 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-31T22:12:57.201 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-08-31T22:12:57.201 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-31T22:12:57.201 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-08-31T22:12:57.201 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-31T22:12:57.201 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-08-31T22:12:57.201 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-31T22:12:57.201 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-31T22:12:57.225 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-08-31T22:12:57.226 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-08-31T22:12:57.226 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-08-31T22:12:57.287 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-08-31T22:12:57.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T22:12:57.338 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: /sbin/restorecon: command not found 2024-08-31T22:12:57.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-08-31T22:12:57.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-31T22:12:57.445+0000 7fc66282f7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-08-31T22:12:57.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-31T22:12:57.445+0000 7fc66282f7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-08-31T22:12:57.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-31T22:12:57.445+0000 7fc66282f7c0 -1 bdev(0x55a15c61e380 /var/lib/ceph/osd/cluster2-0/block) open stat got: (1) Operation not permitted 2024-08-31T22:12:57.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-31T22:12:57.445+0000 7fc66282f7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-08-31T22:13:00.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-08-31T22:13:00.580 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-08-31T22:13:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-31T22:13:00.665+0000 7f89ec2d47c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-08-31T22:13:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-31T22:13:00.665+0000 7f89ec2d47c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-08-31T22:13:00.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-31T22:13:00.669+0000 7f89ec2d47c0 -1 bdev(0x555f958b4380 /var/lib/ceph/osd/cluster2-1/block) open stat got: (1) Operation not permitted 2024-08-31T22:13:00.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-31T22:13:00.669+0000 7f89ec2d47c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-08-31T22:13:03.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-08-31T22:13:03.878 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-08-31T22:13:03.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-31T22:13:03.957+0000 7f34e6b6c7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-08-31T22:13:03.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-31T22:13:03.957+0000 7f34e6b6c7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-08-31T22:13:03.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-31T22:13:03.961+0000 7f34e6b6c7c0 -1 bdev(0x55735d044380 /var/lib/ceph/osd/cluster2-2/block) open stat got: (1) Operation not permitted 2024-08-31T22:13:03.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-31T22:13:03.961+0000 7f34e6b6c7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-08-31T22:13:07.087 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-08-31T22:13:07.141 INFO:tasks.ceph:Reading keys from all nodes... 2024-08-31T22:13:07.141 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:13:07.142 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-08-31T22:13:07.157 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:13:07.157 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-08-31T22:13:07.197 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:13:07.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-08-31T22:13:07.252 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:13:07.252 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-08-31T22:13:07.308 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-08-31T22:13:07.308 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-08-31T22:13:07.315 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:13:07.315 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-08-31T22:13:07.357 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:13:07.357 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-08-31T22:13:07.405 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:13:07.405 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-08-31T22:13:07.453 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:13:07.453 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-08-31T22:13:07.501 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:13:07.501 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-08-31T22:13:07.549 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:13:07.549 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-08-31T22:13:07.597 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:13:07.597 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-08-31T22:13:07.645 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:13:07.645 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-08-31T22:13:07.693 INFO:tasks.ceph:Adding keys to all mons... 2024-08-31T22:13:07.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo tee -a /etc/ceph/cluster2.keyring 2024-08-31T22:13:07.747 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr.x] 2024-08-31T22:13:07.748 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBnldNm88daNRAAXS2O6Vp5RBOjqgi0ukboHg== 2024-08-31T22:13:07.748 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.0] 2024-08-31T22:13:07.748 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBpldNm+mmlGhAALIuBj8pQ+YYV/bsDk35mCA== 2024-08-31T22:13:07.748 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.1] 2024-08-31T22:13:07.748 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBsldNmg7X3JxAAZzggwVxdoBbwD26bDbJn2g== 2024-08-31T22:13:07.748 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.2] 2024-08-31T22:13:07.748 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBvldNmJfFfORAAI62KY6zBrCdDiANJSZxzWA== 2024-08-31T22:13:07.748 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-08-31T22:13:07.748 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBnldNmIDB1NxAAhEFN3c5MS41n+R0yXoffww== 2024-08-31T22:13:07.748 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror] 2024-08-31T22:13:07.748 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBnldNmG25XOhAA+7BPs7vi/CFxGS8T0O497Q== 2024-08-31T22:13:07.748 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.0] 2024-08-31T22:13:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBoldNm6ggOBBAAvMmeZzZ7RGPTwKsj02B+3w== 2024-08-31T22:13:07.749 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.1] 2024-08-31T22:13:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBoldNm1co7CRAAIwufhmnDiq6POc2AUFMsHg== 2024-08-31T22:13:07.749 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.2] 2024-08-31T22:13:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBoldNmKVFhDhAA7p5rgvoYQkm3aGwdO+GrIA== 2024-08-31T22:13:07.749 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.3] 2024-08-31T22:13:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBoldNmsFqrExAAb2lN0jQUImg9CE9BslZvpw== 2024-08-31T22:13:07.749 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.4] 2024-08-31T22:13:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBoldNmaGS6GBAA4jsYMDzRptD6DWJm/tJZ0Q== 2024-08-31T22:13:07.749 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.5] 2024-08-31T22:13:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBoldNmhjndHRAAd5C6yRaZJXI2z8rgt2wJRA== 2024-08-31T22:13:07.749 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.6] 2024-08-31T22:13:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBoldNm04zvIhAA/J27/O6A9+2fHg7wsiERdA== 2024-08-31T22:13:07.750 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-31T22:13:07.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T22:13:07.882 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T22:13:07.970 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T22:13:08.014 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T22:13:08.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T22:13:08.148 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T22:13:08.191 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T22:13:08.274 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T22:13:08.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T22:13:08.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T22:13:08.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T22:13:08.551 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T22:13:08.595 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-08-31T22:13:08.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-08-31T22:13:08.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-08-31T22:13:08.734 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-08-31T22:13:08.744 DEBUG:teuthology.orchestra.run.smithi032:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-08-31T22:13:08.792 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-08-31T22:13:08.793 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-08-31T22:13:08.793 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-08-31T22:13:08.838 INFO:tasks.ceph.cluster2.mon.a:Started 2024-08-31T22:13:08.838 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-08-31T22:13:08.838 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-08-31T22:13:08.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-08-31T22:13:08.841 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-08-31T22:13:08.841 INFO:tasks.ceph:Setting crush tunables to default 2024-08-31T22:13:08.841 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster cluster2 osd crush tunables default 2024-08-31T22:13:09.073 INFO:teuthology.orchestra.run.smithi032.stderr:adjusted tunables profile to default 2024-08-31T22:13:09.091 INFO:tasks.ceph:check_enable_crimson: False 2024-08-31T22:13:09.091 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-08-31T22:13:09.091 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:13:09.091 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-08-31T22:13:09.101 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:13:09.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-08-31T22:13:09.103 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:09.102+0000 7faeb29d9180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-31T22:13:09.154 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:13:09.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-08-31T22:13:09.207 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster cluster2 osd new 2650a43d-a4d7-43a5-81c8-f8512897bad0 0 2024-08-31T22:13:09.458 INFO:teuthology.orchestra.run.smithi032.stdout:0 2024-08-31T22:13:09.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster cluster2 osd new 330b62c6-fe47-4b1e-b61c-7abf9c9fa380 1 2024-08-31T22:13:09.509 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:09.510+0000 7faeb29d9180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-31T22:13:09.648 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:09.650+0000 7faeb29d9180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-31T22:13:09.687 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-31T22:13:09.707 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster cluster2 osd new 47505ad1-7bd0-4a0f-ab31-1cd6778fe9ae 2 2024-08-31T22:13:09.914 INFO:teuthology.orchestra.run.smithi032.stdout:2 2024-08-31T22:13:09.932 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-08-31T22:13:09.932 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-08-31T22:13:09.934 INFO:tasks.ceph.cluster2.osd.0:Started 2024-08-31T22:13:09.935 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-08-31T22:13:09.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-08-31T22:13:09.936 INFO:tasks.ceph.cluster2.osd.1:Started 2024-08-31T22:13:09.937 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-08-31T22:13:09.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-08-31T22:13:09.938 INFO:tasks.ceph.cluster2.osd.2:Started 2024-08-31T22:13:09.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-08-31T22:13:10.082 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:10.082+0000 7faeb29d9180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-31T22:13:10.124 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:13:10.125 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":5,"fsid":"2454aa5a-0087-4f83-8364-979b8131b8af","created":"2024-08-31T22:13:08.964081+0000","modified":"2024-08-31T22:13:09.915598+0000","last_up_change":"0.000000","last_in_change":"2024-08-31T22:13:09.915598+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2650a43d-a4d7-43a5-81c8-f8512897bad0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"330b62c6-fe47-4b1e-b61c-7abf9c9fa380","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47505ad1-7bd0-4a0f-ab31-1cd6778fe9ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T22:13:10.135 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:10.134+0000 7faeb29d9180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-31T22:13:10.142 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-08-31T22:13:10.142 INFO:tasks.ceph:Waiting for OSDs to come up 2024-08-31T22:13:10.186 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:10.186+0000 7faeb29d9180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-31T22:13:10.263 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:10.262+0000 7faeb29d9180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-31T22:13:10.315 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:10.314+0000 7faeb29d9180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-31T22:13:10.391 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:10.390+0000 7faeb29d9180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-31T22:13:10.444 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:10.442+0000 7faeb29d9180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-31T22:13:10.445 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-08-31T22:13:10.588 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:10.590+0000 7faeb29d9180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-31T22:13:10.642 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:10.642+0000 7faeb29d9180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-31T22:13:10.651 INFO:teuthology.misc.health.smithi032.stdout: 2024-08-31T22:13:10.651 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":5,"fsid":"2454aa5a-0087-4f83-8364-979b8131b8af","created":"2024-08-31T22:13:08.964081+0000","modified":"2024-08-31T22:13:09.915598+0000","last_up_change":"0.000000","last_in_change":"2024-08-31T22:13:09.915598+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2650a43d-a4d7-43a5-81c8-f8512897bad0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"330b62c6-fe47-4b1e-b61c-7abf9c9fa380","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47505ad1-7bd0-4a0f-ab31-1cd6778fe9ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T22:13:10.668 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-08-31T22:13:10.866 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-08-31T22:13:10.866 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-08-31T22:13:10.866 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr: from numpy import show_config as show_numpy_config 2024-08-31T22:13:10.874 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:10.874+0000 7faeb29d9180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-31T22:13:11.013 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:11.014+0000 7faeb29d9180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-31T22:13:11.038 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:13:11.037+0000 7fdf8b7bd7c0 -1 Falling back to public interface 2024-08-31T22:13:11.126 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:11.126+0000 7faeb29d9180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-31T22:13:11.190 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:11.190+0000 7faeb29d9180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-31T22:13:11.329 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:11.330+0000 7faeb29d9180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-31T22:13:11.514 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:13:11.513+0000 7f14003617c0 -1 Falling back to public interface 2024-08-31T22:13:11.526 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:13:11.525+0000 7ff2846a67c0 -1 Falling back to public interface 2024-08-31T22:13:11.824 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:11.822+0000 7faeb29d9180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-31T22:13:11.900 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:11.898+0000 7faeb29d9180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-31T22:13:12.050 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:12.050+0000 7faeb29d9180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-31T22:13:12.126 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:12.126+0000 7faeb29d9180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-31T22:13:12.422 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:12.422+0000 7faeb29d9180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-31T22:13:12.478 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:12.478+0000 7faeb29d9180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-31T22:13:13.512 INFO:tasks.ceph.cluster2.mgr.x.smithi007.stderr:2024-08-31T22:13:13.514+0000 7fae59610640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-08-31T22:13:14.977 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:13:14.977+0000 7fdf8b7bd7c0 -1 osd.1 0 log_to_monitors true 2024-08-31T22:13:15.453 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:13:15.453+0000 7f14003617c0 -1 osd.2 0 log_to_monitors true 2024-08-31T22:13:15.658 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:13:15.657+0000 7ff2846a67c0 -1 osd.0 0 log_to_monitors true 2024-08-31T22:13:16.972 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-08-31T22:13:17.457 INFO:teuthology.misc.health.smithi032.stdout: 2024-08-31T22:13:17.457 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":7,"fsid":"2454aa5a-0087-4f83-8364-979b8131b8af","created":"2024-08-31T22:13:08.964081+0000","modified":"2024-08-31T22:13:16.566136+0000","last_up_change":"0.000000","last_in_change":"2024-08-31T22:13:09.915598+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2650a43d-a4d7-43a5-81c8-f8512897bad0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"330b62c6-fe47-4b1e-b61c-7abf9c9fa380","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47505ad1-7bd0-4a0f-ab31-1cd6778fe9ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T22:13:17.479 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-08-31T22:13:17.552 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:13:17.549+0000 7f13fcb1c640 -1 osd.2 0 waiting for initial osdmap 2024-08-31T22:13:17.553 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:13:17.553+0000 7fdf87766640 -1 osd.1 0 waiting for initial osdmap 2024-08-31T22:13:17.573 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:13:17.573+0000 7f13f413f640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-31T22:13:17.588 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:13:17.589+0000 7fdf7f59b640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-31T22:13:18.366 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:13:18.365+0000 7ff280e61640 -1 osd.0 0 waiting for initial osdmap 2024-08-31T22:13:18.385 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:13:18.385+0000 7ff278484640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-31T22:13:23.782 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-08-31T22:13:24.090 INFO:teuthology.misc.health.smithi032.stdout: 2024-08-31T22:13:24.090 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":13,"fsid":"2454aa5a-0087-4f83-8364-979b8131b8af","created":"2024-08-31T22:13:08.964081+0000","modified":"2024-08-31T22:13:22.600398+0000","last_up_change":"2024-08-31T22:13:18.577125+0000","last_in_change":"2024-08-31T22:13:09.915598+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-31T22:13:20.588703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2650a43d-a4d7-43a5-81c8-f8512897bad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3783604114},{"type":"v1","addr":"172.21.15.32:6817","nonce":3783604114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3783604114},{"type":"v1","addr":"172.21.15.32:6819","nonce":3783604114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3783604114},{"type":"v1","addr":"172.21.15.32:6823","nonce":3783604114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3783604114},{"type":"v1","addr":"172.21.15.32:6821","nonce":3783604114}]},"public_addr":"172.21.15.32:6817/3783604114","cluster_addr":"172.21.15.32:6819/3783604114","heartbeat_back_addr":"172.21.15.32:6823/3783604114","heartbeat_front_addr":"172.21.15.32:6821/3783604114","state":["exists","up"]},{"osd":1,"uuid":"330b62c6-fe47-4b1e-b61c-7abf9c9fa380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":201348119},{"type":"v1","addr":"172.21.15.32:6801","nonce":201348119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":201348119},{"type":"v1","addr":"172.21.15.32:6803","nonce":201348119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":201348119},{"type":"v1","addr":"172.21.15.32:6807","nonce":201348119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":201348119},{"type":"v1","addr":"172.21.15.32:6805","nonce":201348119}]},"public_addr":"172.21.15.32:6801/201348119","cluster_addr":"172.21.15.32:6803/201348119","heartbeat_back_addr":"172.21.15.32:6807/201348119","heartbeat_front_addr":"172.21.15.32:6805/201348119","state":["exists","up"]},{"osd":2,"uuid":"47505ad1-7bd0-4a0f-ab31-1cd6778fe9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":4129613497},{"type":"v1","addr":"172.21.15.32:6809","nonce":4129613497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":4129613497},{"type":"v1","addr":"172.21.15.32:6811","nonce":4129613497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":4129613497},{"type":"v1","addr":"172.21.15.32:6815","nonce":4129613497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":4129613497},{"type":"v1","addr":"172.21.15.32:6813","nonce":4129613497}]},"public_addr":"172.21.15.32:6809/4129613497","cluster_addr":"172.21.15.32:6811/4129613497","heartbeat_back_addr":"172.21.15.32:6815/4129613497","heartbeat_front_addr":"172.21.15.32:6813/4129613497","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:13:16.673817+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:13:15.983995+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:13:16.448905+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T22:13:24.108 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-08-31T22:13:24.108 INFO:tasks.ceph:Creating RBD pool 2024-08-31T22:13:24.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-08-31T22:13:24.614 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'rbd' created 2024-08-31T22:13:24.643 DEBUG:teuthology.orchestra.run.smithi032:> rbd --cluster cluster2 pool init rbd 2024-08-31T22:13:27.647 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-08-31T22:13:27.647 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-08-31T22:13:27.647 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-08-31T22:13:27.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-08-31T22:13:27.648 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-08-31T22:13:28.003 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:13:28.020 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":474455981},{"type":"v1","addr":"172.21.15.7:6827","nonce":474455981}]},"active_addr":"172.21.15.7:6827/474455981","active_change":"2024-08-31T22:13:12.500695+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":1246353491}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":2969549406}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":356717902}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":3232295469}]}]} 2024-08-31T22:13:28.022 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-08-31T22:13:28.022 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-08-31T22:13:28.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-08-31T22:13:28.314 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:13:28.314 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"fsid":"2454aa5a-0087-4f83-8364-979b8131b8af","created":"2024-08-31T22:13:08.964081+0000","modified":"2024-08-31T22:13:27.628027+0000","last_up_change":"2024-08-31T22:13:18.577125+0000","last_in_change":"2024-08-31T22:13:09.915598+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-31T22:13:20.588703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-31T22:13:24.457840+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2650a43d-a4d7-43a5-81c8-f8512897bad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3783604114},{"type":"v1","addr":"172.21.15.32:6817","nonce":3783604114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3783604114},{"type":"v1","addr":"172.21.15.32:6819","nonce":3783604114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3783604114},{"type":"v1","addr":"172.21.15.32:6823","nonce":3783604114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3783604114},{"type":"v1","addr":"172.21.15.32:6821","nonce":3783604114}]},"public_addr":"172.21.15.32:6817/3783604114","cluster_addr":"172.21.15.32:6819/3783604114","heartbeat_back_addr":"172.21.15.32:6823/3783604114","heartbeat_front_addr":"172.21.15.32:6821/3783604114","state":["exists","up"]},{"osd":1,"uuid":"330b62c6-fe47-4b1e-b61c-7abf9c9fa380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":201348119},{"type":"v1","addr":"172.21.15.32:6801","nonce":201348119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":201348119},{"type":"v1","addr":"172.21.15.32:6803","nonce":201348119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":201348119},{"type":"v1","addr":"172.21.15.32:6807","nonce":201348119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":201348119},{"type":"v1","addr":"172.21.15.32:6805","nonce":201348119}]},"public_addr":"172.21.15.32:6801/201348119","cluster_addr":"172.21.15.32:6803/201348119","heartbeat_back_addr":"172.21.15.32:6807/201348119","heartbeat_front_addr":"172.21.15.32:6805/201348119","state":["exists","up"]},{"osd":2,"uuid":"47505ad1-7bd0-4a0f-ab31-1cd6778fe9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":4129613497},{"type":"v1","addr":"172.21.15.32:6809","nonce":4129613497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":4129613497},{"type":"v1","addr":"172.21.15.32:6811","nonce":4129613497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":4129613497},{"type":"v1","addr":"172.21.15.32:6815","nonce":4129613497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":4129613497},{"type":"v1","addr":"172.21.15.32:6813","nonce":4129613497}]},"public_addr":"172.21.15.32:6809/4129613497","cluster_addr":"172.21.15.32:6811/4129613497","heartbeat_back_addr":"172.21.15.32:6815/4129613497","heartbeat_front_addr":"172.21.15.32:6813/4129613497","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:13:16.673817+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:13:15.983995+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:13:16.448905+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T22:13:28.331 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-08-31T22:13:28.331 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-08-31T22:13:28.638 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:13:28.638 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"fsid":"2454aa5a-0087-4f83-8364-979b8131b8af","created":"2024-08-31T22:13:08.964081+0000","modified":"2024-08-31T22:13:27.628027+0000","last_up_change":"2024-08-31T22:13:18.577125+0000","last_in_change":"2024-08-31T22:13:09.915598+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-31T22:13:20.588703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-31T22:13:24.457840+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2650a43d-a4d7-43a5-81c8-f8512897bad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3783604114},{"type":"v1","addr":"172.21.15.32:6817","nonce":3783604114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3783604114},{"type":"v1","addr":"172.21.15.32:6819","nonce":3783604114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3783604114},{"type":"v1","addr":"172.21.15.32:6823","nonce":3783604114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3783604114},{"type":"v1","addr":"172.21.15.32:6821","nonce":3783604114}]},"public_addr":"172.21.15.32:6817/3783604114","cluster_addr":"172.21.15.32:6819/3783604114","heartbeat_back_addr":"172.21.15.32:6823/3783604114","heartbeat_front_addr":"172.21.15.32:6821/3783604114","state":["exists","up"]},{"osd":1,"uuid":"330b62c6-fe47-4b1e-b61c-7abf9c9fa380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":201348119},{"type":"v1","addr":"172.21.15.32:6801","nonce":201348119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":201348119},{"type":"v1","addr":"172.21.15.32:6803","nonce":201348119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":201348119},{"type":"v1","addr":"172.21.15.32:6807","nonce":201348119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":201348119},{"type":"v1","addr":"172.21.15.32:6805","nonce":201348119}]},"public_addr":"172.21.15.32:6801/201348119","cluster_addr":"172.21.15.32:6803/201348119","heartbeat_back_addr":"172.21.15.32:6807/201348119","heartbeat_front_addr":"172.21.15.32:6805/201348119","state":["exists","up"]},{"osd":2,"uuid":"47505ad1-7bd0-4a0f-ab31-1cd6778fe9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":4129613497},{"type":"v1","addr":"172.21.15.32:6809","nonce":4129613497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":4129613497},{"type":"v1","addr":"172.21.15.32:6811","nonce":4129613497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":4129613497},{"type":"v1","addr":"172.21.15.32:6815","nonce":4129613497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":4129613497},{"type":"v1","addr":"172.21.15.32:6813","nonce":4129613497}]},"public_addr":"172.21.15.32:6809/4129613497","cluster_addr":"172.21.15.32:6811/4129613497","heartbeat_back_addr":"172.21.15.32:6815/4129613497","heartbeat_front_addr":"172.21.15.32:6813/4129613497","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:13:16.673817+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:13:15.983995+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:13:16.448905+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T22:13:28.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-08-31T22:13:28.658 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-08-31T22:13:28.658 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-08-31T22:13:28.812 INFO:teuthology.orchestra.run.smithi032.stdout:38654705669 2024-08-31T22:13:28.813 INFO:teuthology.orchestra.run.smithi032.stdout:38654705670 2024-08-31T22:13:28.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-08-31T22:13:28.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-08-31T22:13:28.815 INFO:teuthology.orchestra.run.smithi032.stdout:38654705669 2024-08-31T22:13:28.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-08-31T22:13:29.146 INFO:teuthology.orchestra.run.smithi032.stdout:38654705669 2024-08-31T22:13:29.163 INFO:teuthology.orchestra.run.smithi032.stdout:38654705668 2024-08-31T22:13:29.169 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705670 got 38654705669 for osd.2 2024-08-31T22:13:29.180 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705668 for osd.1 2024-08-31T22:13:29.212 INFO:teuthology.orchestra.run.smithi032.stdout:38654705668 2024-08-31T22:13:29.228 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705668 for osd.0 2024-08-31T22:13:30.171 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-08-31T22:13:30.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-08-31T22:13:30.229 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-08-31T22:13:30.489 INFO:teuthology.orchestra.run.smithi032.stdout:38654705669 2024-08-31T22:13:30.513 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705670 got 38654705669 for osd.2 2024-08-31T22:13:30.515 INFO:teuthology.orchestra.run.smithi032.stdout:38654705668 2024-08-31T22:13:30.532 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705668 for osd.1 2024-08-31T22:13:30.569 INFO:teuthology.orchestra.run.smithi032.stdout:38654705668 2024-08-31T22:13:30.586 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705668 for osd.0 2024-08-31T22:13:31.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-08-31T22:13:31.533 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-08-31T22:13:31.587 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-08-31T22:13:31.839 INFO:teuthology.orchestra.run.smithi032.stdout:38654705671 2024-08-31T22:13:31.857 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705670 got 38654705671 for osd.2 2024-08-31T22:13:31.857 DEBUG:teuthology.parallel:result is None 2024-08-31T22:13:31.872 INFO:teuthology.orchestra.run.smithi032.stdout:38654705670 2024-08-31T22:13:31.889 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705670 for osd.1 2024-08-31T22:13:31.889 DEBUG:teuthology.parallel:result is None 2024-08-31T22:13:31.900 INFO:teuthology.orchestra.run.smithi032.stdout:38654705670 2024-08-31T22:13:31.917 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705670 for osd.0 2024-08-31T22:13:31.917 DEBUG:teuthology.parallel:result is None 2024-08-31T22:13:31.917 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-08-31T22:13:31.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-08-31T22:13:32.249 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:13:32.249 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-31T22:13:32.266 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-08-31T22:13:30.511267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":94124,"kb_used_data":552,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283021396,"statfs":{"total":289910292480,"available":289813909504,"internally_reserved":0,"allocated":565248,"data_stored":1309576,"data_compressed":6354,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.892755"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:13:27.638168+0000","last_change":"2024-08-31T22:13:27.638168+0000","last_active":"2024-08-31T22:13:27.638168+0000","last_peered":"2024-08-31T22:13:27.638168+0000","last_clean":"2024-08-31T22:13:27.638168+0000","last_became_active":"2024-08-31T22:13:25.634194+0000","last_became_peered":"2024-08-31T22:13:25.634194+0000","last_unstale":"2024-08-31T22:13:27.638168+0000","last_undegraded":"2024-08-31T22:13:27.638168+0000","last_fullsized":"2024-08-31T22:13:27.638168+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T03:43:12.617984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024029400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:13:27.636754+0000","last_change":"2024-08-31T22:13:27.636754+0000","last_active":"2024-08-31T22:13:27.636754+0000","last_peered":"2024-08-31T22:13:27.636754+0000","last_clean":"2024-08-31T22:13:27.636754+0000","last_became_active":"2024-08-31T22:13:25.634366+0000","last_became_peered":"2024-08-31T22:13:25.634366+0000","last_unstale":"2024-08-31T22:13:27.636754+0000","last_undegraded":"2024-08-31T22:13:27.636754+0000","last_fullsized":"2024-08-31T22:13:27.636754+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:32:00.532608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001106084,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:13:27.637651+0000","last_change":"2024-08-31T22:13:27.637651+0000","last_active":"2024-08-31T22:13:27.637651+0000","last_peered":"2024-08-31T22:13:27.637651+0000","last_clean":"2024-08-31T22:13:27.637651+0000","last_became_active":"2024-08-31T22:13:25.634100+0000","last_became_peered":"2024-08-31T22:13:25.634100+0000","last_unstale":"2024-08-31T22:13:27.637651+0000","last_undegraded":"2024-08-31T22:13:27.637651+0000","last_fullsized":"2024-08-31T22:13:27.637651+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T09:42:20.559113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001870128,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:13:27.637511+0000","last_change":"2024-08-31T22:13:27.637511+0000","last_active":"2024-08-31T22:13:27.637511+0000","last_peered":"2024-08-31T22:13:27.637511+0000","last_clean":"2024-08-31T22:13:27.637511+0000","last_became_active":"2024-08-31T22:13:25.628362+0000","last_became_peered":"2024-08-31T22:13:25.628362+0000","last_unstale":"2024-08-31T22:13:27.637511+0000","last_undegraded":"2024-08-31T22:13:27.637511+0000","last_fullsized":"2024-08-31T22:13:27.637511+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T07:08:57.946059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001773806,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"17'2","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:13:27.641425+0000","last_change":"2024-08-31T22:13:27.640074+0000","last_active":"2024-08-31T22:13:27.641425+0000","last_peered":"2024-08-31T22:13:27.641425+0000","last_clean":"2024-08-31T22:13:27.641425+0000","last_became_active":"2024-08-31T22:13:25.628711+0000","last_became_peered":"2024-08-31T22:13:25.628711+0000","last_unstale":"2024-08-31T22:13:27.641425+0000","last_undegraded":"2024-08-31T22:13:27.641425+0000","last_fullsized":"2024-08-31T22:13:27.641425+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T08:42:16.373330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002200956,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:13:27.762539+0000","last_change":"2024-08-31T22:13:27.762539+0000","last_active":"2024-08-31T22:13:27.762539+0000","last_peered":"2024-08-31T22:13:27.762539+0000","last_clean":"2024-08-31T22:13:27.762539+0000","last_became_active":"2024-08-31T22:13:25.627201+0000","last_became_peered":"2024-08-31T22:13:25.627201+0000","last_unstale":"2024-08-31T22:13:27.762539+0000","last_undegraded":"2024-08-31T22:13:27.762539+0000","last_fullsized":"2024-08-31T22:13:27.762539+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T09:46:26.013807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054468599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:13:27.762844+0000","last_change":"2024-08-31T22:13:27.762844+0000","last_active":"2024-08-31T22:13:27.762844+0000","last_peered":"2024-08-31T22:13:27.762844+0000","last_clean":"2024-08-31T22:13:27.762844+0000","last_became_active":"2024-08-31T22:13:25.631027+0000","last_became_peered":"2024-08-31T22:13:25.631027+0000","last_unstale":"2024-08-31T22:13:27.762844+0000","last_undegraded":"2024-08-31T22:13:27.762844+0000","last_fullsized":"2024-08-31T22:13:27.762844+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:16:59.947661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060313200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:13:27.636882+0000","last_change":"2024-08-31T22:13:27.636882+0000","last_active":"2024-08-31T22:13:27.636882+0000","last_peered":"2024-08-31T22:13:27.636882+0000","last_clean":"2024-08-31T22:13:27.636882+0000","last_became_active":"2024-08-31T22:13:25.628867+0000","last_became_peered":"2024-08-31T22:13:25.628867+0000","last_unstale":"2024-08-31T22:13:27.636882+0000","last_undegraded":"2024-08-31T22:13:27.636882+0000","last_fullsized":"2024-08-31T22:13:27.636882+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:00:38.825010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001191278,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'97","reported_seq":147,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:13:27.635534+0000","last_change":"2024-08-31T22:13:21.644125+0000","last_active":"2024-08-31T22:13:27.635534+0000","last_peered":"2024-08-31T22:13:27.635534+0000","last_clean":"2024-08-31T22:13:27.635534+0000","last_became_active":"2024-08-31T22:13:21.641333+0000","last_became_peered":"2024-08-31T22:13:21.641333+0000","last_unstale":"2024-08-31T22:13:27.635534+0000","last_undegraded":"2024-08-31T22:13:27.635534+0000","last_fullsized":"2024-08-31T22:13:27.635534+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:20.589879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:20.589879+0000","last_clean_scrub_stamp":"2024-08-31T22:13:20.589879+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:38:29.946497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6354,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31332,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340508,"statfs":{"total":96636764160,"available":96604680192,"internally_reserved":0,"allocated":147456,"data_stored":42934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":208896,"data_stored":633321,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":208896,"data_stored":633321,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T22:13:32.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-08-31T22:13:32.563 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:13:32.564 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-31T22:13:32.582 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-08-31T22:13:32.511793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":94124,"kb_used_data":552,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283021396,"statfs":{"total":289910292480,"available":289813909504,"internally_reserved":0,"allocated":565248,"data_stored":1309576,"data_compressed":6354,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.894012"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:13:27.638168+0000","last_change":"2024-08-31T22:13:27.638168+0000","last_active":"2024-08-31T22:13:27.638168+0000","last_peered":"2024-08-31T22:13:27.638168+0000","last_clean":"2024-08-31T22:13:27.638168+0000","last_became_active":"2024-08-31T22:13:25.634194+0000","last_became_peered":"2024-08-31T22:13:25.634194+0000","last_unstale":"2024-08-31T22:13:27.638168+0000","last_undegraded":"2024-08-31T22:13:27.638168+0000","last_fullsized":"2024-08-31T22:13:27.638168+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T03:43:12.617984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024029400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:13:27.636754+0000","last_change":"2024-08-31T22:13:27.636754+0000","last_active":"2024-08-31T22:13:27.636754+0000","last_peered":"2024-08-31T22:13:27.636754+0000","last_clean":"2024-08-31T22:13:27.636754+0000","last_became_active":"2024-08-31T22:13:25.634366+0000","last_became_peered":"2024-08-31T22:13:25.634366+0000","last_unstale":"2024-08-31T22:13:27.636754+0000","last_undegraded":"2024-08-31T22:13:27.636754+0000","last_fullsized":"2024-08-31T22:13:27.636754+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:32:00.532608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001106084,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:13:27.637651+0000","last_change":"2024-08-31T22:13:27.637651+0000","last_active":"2024-08-31T22:13:27.637651+0000","last_peered":"2024-08-31T22:13:27.637651+0000","last_clean":"2024-08-31T22:13:27.637651+0000","last_became_active":"2024-08-31T22:13:25.634100+0000","last_became_peered":"2024-08-31T22:13:25.634100+0000","last_unstale":"2024-08-31T22:13:27.637651+0000","last_undegraded":"2024-08-31T22:13:27.637651+0000","last_fullsized":"2024-08-31T22:13:27.637651+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T09:42:20.559113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001870128,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:13:27.637511+0000","last_change":"2024-08-31T22:13:27.637511+0000","last_active":"2024-08-31T22:13:27.637511+0000","last_peered":"2024-08-31T22:13:27.637511+0000","last_clean":"2024-08-31T22:13:27.637511+0000","last_became_active":"2024-08-31T22:13:25.628362+0000","last_became_peered":"2024-08-31T22:13:25.628362+0000","last_unstale":"2024-08-31T22:13:27.637511+0000","last_undegraded":"2024-08-31T22:13:27.637511+0000","last_fullsized":"2024-08-31T22:13:27.637511+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T07:08:57.946059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001773806,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"17'2","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:13:27.641425+0000","last_change":"2024-08-31T22:13:27.640074+0000","last_active":"2024-08-31T22:13:27.641425+0000","last_peered":"2024-08-31T22:13:27.641425+0000","last_clean":"2024-08-31T22:13:27.641425+0000","last_became_active":"2024-08-31T22:13:25.628711+0000","last_became_peered":"2024-08-31T22:13:25.628711+0000","last_unstale":"2024-08-31T22:13:27.641425+0000","last_undegraded":"2024-08-31T22:13:27.641425+0000","last_fullsized":"2024-08-31T22:13:27.641425+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T08:42:16.373330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002200956,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:13:27.762539+0000","last_change":"2024-08-31T22:13:27.762539+0000","last_active":"2024-08-31T22:13:27.762539+0000","last_peered":"2024-08-31T22:13:27.762539+0000","last_clean":"2024-08-31T22:13:27.762539+0000","last_became_active":"2024-08-31T22:13:25.627201+0000","last_became_peered":"2024-08-31T22:13:25.627201+0000","last_unstale":"2024-08-31T22:13:27.762539+0000","last_undegraded":"2024-08-31T22:13:27.762539+0000","last_fullsized":"2024-08-31T22:13:27.762539+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T09:46:26.013807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054468599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:13:27.762844+0000","last_change":"2024-08-31T22:13:27.762844+0000","last_active":"2024-08-31T22:13:27.762844+0000","last_peered":"2024-08-31T22:13:27.762844+0000","last_clean":"2024-08-31T22:13:27.762844+0000","last_became_active":"2024-08-31T22:13:25.631027+0000","last_became_peered":"2024-08-31T22:13:25.631027+0000","last_unstale":"2024-08-31T22:13:27.762844+0000","last_undegraded":"2024-08-31T22:13:27.762844+0000","last_fullsized":"2024-08-31T22:13:27.762844+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:16:59.947661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060313200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:13:27.636882+0000","last_change":"2024-08-31T22:13:27.636882+0000","last_active":"2024-08-31T22:13:27.636882+0000","last_peered":"2024-08-31T22:13:27.636882+0000","last_clean":"2024-08-31T22:13:27.636882+0000","last_became_active":"2024-08-31T22:13:25.628867+0000","last_became_peered":"2024-08-31T22:13:25.628867+0000","last_unstale":"2024-08-31T22:13:27.636882+0000","last_undegraded":"2024-08-31T22:13:27.636882+0000","last_fullsized":"2024-08-31T22:13:27.636882+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:00:38.825010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001191278,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'97","reported_seq":147,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-08-31T22:13:27.635534+0000","last_change":"2024-08-31T22:13:21.644125+0000","last_active":"2024-08-31T22:13:27.635534+0000","last_peered":"2024-08-31T22:13:27.635534+0000","last_clean":"2024-08-31T22:13:27.635534+0000","last_became_active":"2024-08-31T22:13:21.641333+0000","last_became_peered":"2024-08-31T22:13:21.641333+0000","last_unstale":"2024-08-31T22:13:27.635534+0000","last_undegraded":"2024-08-31T22:13:27.635534+0000","last_fullsized":"2024-08-31T22:13:27.635534+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:20.589879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:20.589879+0000","last_clean_scrub_stamp":"2024-08-31T22:13:20.589879+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:38:29.946497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6354,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31332,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340508,"statfs":{"total":96636764160,"available":96604680192,"internally_reserved":0,"allocated":147456,"data_stored":42934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":208896,"data_stored":633321,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":208896,"data_stored":633321,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T22:13:32.583 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-08-31T22:13:32.583 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-08-31T22:13:32.583 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-08-31T22:13:32.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-08-31T22:13:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:13:32.921 INFO:teuthology.orchestra.run.smithi032.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-31T22:13:32.940 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-08-31T22:13:32.940 INFO:teuthology.run_tasks:Running task exec... 2024-08-31T22:13:32.950 INFO:teuthology.task.exec:Executing custom commands... 2024-08-31T22:13:32.950 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi032.front.sepia.ceph.com 2024-08-31T22:13:32.951 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-08-31T22:13:33.320 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.4] 2024-08-31T22:13:33.320 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBCldNmhPI5NRAAZhTZqRhPHUOCsvo7eY7cqg== 2024-08-31T22:13:33.320 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "profile rbd-mirror" 2024-08-31T22:13:33.321 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "profile rbd" 2024-08-31T22:13:33.321 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.mirror.4 2024-08-31T22:13:33.340 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-08-31T22:13:33.725 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.5] 2024-08-31T22:13:33.725 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBCldNmNLbPORAAPCiMrYgb4ZFp8I+ccMR4Dg== 2024-08-31T22:13:33.726 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "profile rbd-mirror" 2024-08-31T22:13:33.726 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "profile rbd" 2024-08-31T22:13:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.mirror.5 2024-08-31T22:13:33.745 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-08-31T22:13:34.127 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.6] 2024-08-31T22:13:34.127 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBDldNm6gOKAxAAHZ8F3h9oEm7xqCozbtMpvg== 2024-08-31T22:13:34.127 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "profile rbd-mirror" 2024-08-31T22:13:34.127 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "profile rbd" 2024-08-31T22:13:34.127 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.mirror.6 2024-08-31T22:13:34.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-08-31T22:13:34.512 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.4] 2024-08-31T22:13:34.512 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBoldNmaGS6GBAA4jsYMDzRptD6DWJm/tJZ0Q== 2024-08-31T22:13:34.512 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "profile rbd-mirror" 2024-08-31T22:13:34.512 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "profile rbd" 2024-08-31T22:13:34.513 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.mirror.4 2024-08-31T22:13:34.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-08-31T22:13:34.907 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.5] 2024-08-31T22:13:34.907 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBoldNmhjndHRAAd5C6yRaZJXI2z8rgt2wJRA== 2024-08-31T22:13:34.907 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "profile rbd-mirror" 2024-08-31T22:13:34.907 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "profile rbd" 2024-08-31T22:13:34.908 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.mirror.5 2024-08-31T22:13:34.927 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-08-31T22:13:35.286 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.6] 2024-08-31T22:13:35.287 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBoldNm04zvIhAA/J27/O6A9+2fHg7wsiERdA== 2024-08-31T22:13:35.287 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "profile rbd-mirror" 2024-08-31T22:13:35.287 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "profile rbd" 2024-08-31T22:13:35.287 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.mirror.6 2024-08-31T22:13:35.305 INFO:teuthology.run_tasks:Running task exec... 2024-08-31T22:13:35.322 INFO:teuthology.task.exec:Executing custom commands... 2024-08-31T22:13:35.322 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi032.front.sepia.ceph.com 2024-08-31T22:13:35.322 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-08-31T22:13:35.695 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror] 2024-08-31T22:13:35.695 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBCldNmZvZgGhAABvbh6KrOP92vspWT+djHFA== 2024-08-31T22:13:35.695 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "profile rbd-mirror-peer" 2024-08-31T22:13:35.695 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "profile rbd" 2024-08-31T22:13:35.695 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.mirror 2024-08-31T22:13:35.713 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-08-31T22:13:36.106 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.0] 2024-08-31T22:13:36.106 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBCldNmIRaZIBAA5D/O78xVRqMaWq5i1ALKDg== 2024-08-31T22:13:36.106 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "profile rbd-mirror" 2024-08-31T22:13:36.106 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "profile rbd" 2024-08-31T22:13:36.107 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.mirror.0 2024-08-31T22:13:36.126 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-08-31T22:13:36.500 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.1] 2024-08-31T22:13:36.501 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBCldNmdN2SJRAAqIIJXHJsL8PYPso06bmeJw== 2024-08-31T22:13:36.501 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "profile rbd-mirror" 2024-08-31T22:13:36.501 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "profile rbd" 2024-08-31T22:13:36.501 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.mirror.1 2024-08-31T22:13:36.518 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-08-31T22:13:36.879 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.2] 2024-08-31T22:13:36.879 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBCldNmUUTJKhAAtWLMhzIEEgMrZfO2RR1cPw== 2024-08-31T22:13:36.879 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "profile rbd-mirror" 2024-08-31T22:13:36.879 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "profile rbd" 2024-08-31T22:13:36.880 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.mirror.2 2024-08-31T22:13:36.898 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-08-31T22:13:37.269 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.3] 2024-08-31T22:13:37.269 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBCldNm4QOsLxAAYycFBGOT5qMylr2tDf5gVQ== 2024-08-31T22:13:37.269 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "profile rbd-mirror" 2024-08-31T22:13:37.269 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "profile rbd" 2024-08-31T22:13:37.269 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.mirror.3 2024-08-31T22:13:37.287 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-08-31T22:13:37.655 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror] 2024-08-31T22:13:37.655 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBnldNmG25XOhAA+7BPs7vi/CFxGS8T0O497Q== 2024-08-31T22:13:37.655 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "profile rbd-mirror-peer" 2024-08-31T22:13:37.655 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "profile rbd" 2024-08-31T22:13:37.655 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.mirror 2024-08-31T22:13:37.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-08-31T22:13:38.036 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.0] 2024-08-31T22:13:38.036 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBoldNm6ggOBBAAvMmeZzZ7RGPTwKsj02B+3w== 2024-08-31T22:13:38.037 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "profile rbd-mirror" 2024-08-31T22:13:38.037 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "profile rbd" 2024-08-31T22:13:38.037 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.mirror.0 2024-08-31T22:13:38.056 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-08-31T22:13:38.428 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.1] 2024-08-31T22:13:38.428 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBoldNm1co7CRAAIwufhmnDiq6POc2AUFMsHg== 2024-08-31T22:13:38.428 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "profile rbd-mirror" 2024-08-31T22:13:38.429 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "profile rbd" 2024-08-31T22:13:38.429 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.mirror.1 2024-08-31T22:13:38.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-08-31T22:13:38.819 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.2] 2024-08-31T22:13:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBoldNmKVFhDhAA7p5rgvoYQkm3aGwdO+GrIA== 2024-08-31T22:13:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "profile rbd-mirror" 2024-08-31T22:13:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "profile rbd" 2024-08-31T22:13:38.819 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.mirror.2 2024-08-31T22:13:38.838 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-08-31T22:13:39.214 INFO:teuthology.orchestra.run.smithi032.stdout:[client.mirror.3] 2024-08-31T22:13:39.214 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBoldNmsFqrExAAb2lN0jQUImg9CE9BslZvpw== 2024-08-31T22:13:39.214 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "profile rbd-mirror" 2024-08-31T22:13:39.214 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "profile rbd" 2024-08-31T22:13:39.215 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.mirror.3 2024-08-31T22:13:39.234 INFO:teuthology.run_tasks:Running task workunit... 2024-08-31T22:13:39.251 INFO:tasks.workunit:Pulling workunits from ref 3091dfc0187f68df2afd4e392dc4c381cfdc1cad 2024-08-31T22:13:39.251 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-31T22:13:39.251 DEBUG:teuthology.orchestra.run.smithi032:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-08-31T22:13:39.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T22:13:39.257 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-08-31T22:13:39.257 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-08-31T22:13:39.304 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-08-31T22:13:39.304 DEBUG:teuthology.orchestra.run.smithi032:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-08-31T22:13:39.352 INFO:tasks.workunit:timeout=3h 2024-08-31T22:13:39.352 INFO:tasks.workunit:cleanup=True 2024-08-31T22:13:39.353 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 3091dfc0187f68df2afd4e392dc4c381cfdc1cad 2024-08-31T22:13:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-08-31T22:14:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Updating files: 93% (12056/12888) Updating files: 94% (12115/12888) Updating files: 95% (12244/12888) Updating files: 96% (12373/12888) Updating files: 97% (12502/12888) Updating files: 98% (12631/12888) Updating files: 99% (12760/12888) Updating files: 100% (12888/12888) Updating files: 100% (12888/12888), done. 2024-08-31T22:14:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Note: switching to '3091dfc0187f68df2afd4e392dc4c381cfdc1cad'. 2024-08-31T22:14:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: 2024-08-31T22:14:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-08-31T22:14:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:changes and commit them, and you can discard any commits you make in this 2024-08-31T22:14:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:state without impacting any branches by switching back to a branch. 2024-08-31T22:14:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: 2024-08-31T22:14:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:If you want to create a new branch to retain commits you create, you may 2024-08-31T22:14:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:do so (now or later) by using -c with the switch command. Example: 2024-08-31T22:14:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: 2024-08-31T22:14:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: git switch -c 2024-08-31T22:14:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: 2024-08-31T22:14:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Or undo this operation with: 2024-08-31T22:14:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: 2024-08-31T22:14:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: git switch - 2024-08-31T22:14:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: 2024-08-31T22:14:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-08-31T22:14:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: 2024-08-31T22:14:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:HEAD is now at 3091dfc0187 Merge pull request #59540 from zdover23/wip-doc-2024-08-31-backport-59528-to-squid 2024-08-31T22:14:33.240 DEBUG:teuthology.orchestra.run.smithi032:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-08-31T22:14:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-08-31T22:14:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-08-31T22:14:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-08-31T22:14:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-08-31T22:14:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-08-31T22:14:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-08-31T22:14:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-08-31T22:14:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-08-31T22:14:34.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-08-31T22:14:34.221 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-31T22:14:34.221 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-08-31T22:14:34.271 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-08-31T22:14:34.272 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-08-31T22:14:34.272 DEBUG:teuthology.orchestra.run.smithi032:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3091dfc0187f68df2afd4e392dc4c381cfdc1cad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' RBD_MIRROR_MIN_COMPAT_CLIENT=octopus RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-08-31T22:14:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ IMAGE_COUNT=50 2024-08-31T22:14:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ export LOCKDEP=0 2024-08-31T22:14:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ LOCKDEP=0 2024-08-31T22:14:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-08-31T22:14:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-08-31T22:14:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ type xmlstarlet 2024-08-31T22:14:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ XMLSTARLET=xmlstarlet 2024-08-31T22:14:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ RBD_MIRROR_INSTANCES=2 2024-08-31T22:14:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CLUSTER1=cluster1 2024-08-31T22:14:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CLUSTER2=cluster2 2024-08-31T22:14:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ PEER_CLUSTER_SUFFIX= 2024-08-31T22:14:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ POOL=mirror 2024-08-31T22:14:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ PARENT_POOL=mirror_parent 2024-08-31T22:14:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ NS1=ns1 2024-08-31T22:14:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ NS2=ns2 2024-08-31T22:14:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ TEMPDIR= 2024-08-31T22:14:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ID=mirror 2024-08-31T22:14:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,journaling 2024-08-31T22:14:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-08-31T22:14:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ RBD_MIRROR_MODE=journal 2024-08-31T22:14:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ MIRROR_POOL_MODE=pool 2024-08-31T22:14:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' journal = snapshot ']' 2024-08-31T22:14:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-08-31T22:14:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS='--id mirror' 2024-08-31T22:14:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ LAST_MIRROR_INSTANCE=1 2024-08-31T22:14:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-08-31T22:14:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-08-31T22:14:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-08-31T22:14:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_BIN=. 2024-08-31T22:14:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_SRC=. 2024-08-31T22:14:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' -e CMakeCache.txt ']' 2024-08-31T22:14:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 -gt 0 ']' 2024-08-31T22:14:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ setup 2024-08-31T22:14:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local c 2024-08-31T22:14:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-08-31T22:14:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ setup_tempdir 2024-08-31T22:14:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:14:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ mktemp -d 2024-08-31T22:14:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ TEMPDIR=/tmp/tmp.iYfzsjLopX 2024-08-31T22:14:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -z 1 ']' 2024-08-31T22:14:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ setup_pools cluster1 cluster2 2024-08-31T22:14:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:14:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local remote_cluster=cluster2 2024-08-31T22:14:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mon_map_file 2024-08-31T22:14:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mon_addr 2024-08-31T22:14:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local admin_key_file 2024-08-31T22:14:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local uuid 2024-08-31T22:14:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ CEPH_ARGS= 2024-08-31T22:14:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-08-31T22:14:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:pool 'mirror' already exists 2024-08-31T22:14:35.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ CEPH_ARGS= 2024-08-31T22:14:35.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-08-31T22:14:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:pool 'mirror_parent' already exists 2024-08-31T22:14:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ CEPH_ARGS= 2024-08-31T22:14:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 pool init mirror 2024-08-31T22:14:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ CEPH_ARGS= 2024-08-31T22:14:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-08-31T22:14:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:14:42.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ CEPH_ARGS= 2024-08-31T22:14:42.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror pool 2024-08-31T22:14:42.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-08-31T22:14:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-08-31T22:14:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-08-31T22:14:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 pool 2024-08-31T22:14:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-08-31T22:14:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -z ']' 2024-08-31T22:14:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -z ']' 2024-08-31T22:14:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ peer_add cluster1 mirror cluster2 2024-08-31T22:14:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:14:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:14:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local client_cluster=cluster2 2024-08-31T22:14:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local remote_cluster=cluster2 2024-08-31T22:14:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local uuid_var_name 2024-08-31T22:14:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:14:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local error_code 2024-08-31T22:14:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local peer_uuid 2024-08-31T22:14:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 16 32 2024-08-31T22:14:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:14:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-08-31T22:14:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ peer_uuid=53e29070-4ccf-493a-8bc9-b885923d1098 2024-08-31T22:14:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=0 2024-08-31T22:14:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:14:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 -eq 17 ']' 2024-08-31T22:14:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 0 -eq 0 2024-08-31T22:14:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:14:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:14:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-08-31T22:14:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:14:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror_parent 2024-08-31T22:14:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local client_cluster=cluster2 2024-08-31T22:14:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local remote_cluster=cluster2 2024-08-31T22:14:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local uuid_var_name 2024-08-31T22:14:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:14:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local error_code 2024-08-31T22:14:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local peer_uuid 2024-08-31T22:14:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 16 32 2024-08-31T22:14:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:14:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-08-31T22:14:42.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ peer_uuid=65dd1852-8f9e-4b84-9a2b-d9fbddef3d48 2024-08-31T22:14:42.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=0 2024-08-31T22:14:42.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:14:42.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 -eq 17 ']' 2024-08-31T22:14:42.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 0 -eq 0 2024-08-31T22:14:42.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:14:42.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:14:42.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ setup_pools cluster2 cluster1 2024-08-31T22:14:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:14:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local remote_cluster=cluster1 2024-08-31T22:14:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mon_map_file 2024-08-31T22:14:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mon_addr 2024-08-31T22:14:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local admin_key_file 2024-08-31T22:14:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local uuid 2024-08-31T22:14:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ CEPH_ARGS= 2024-08-31T22:14:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-08-31T22:14:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:pool 'mirror' already exists 2024-08-31T22:14:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ CEPH_ARGS= 2024-08-31T22:14:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-08-31T22:14:45.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:pool 'mirror_parent' already exists 2024-08-31T22:14:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ CEPH_ARGS= 2024-08-31T22:14:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 pool init mirror 2024-08-31T22:14:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ CEPH_ARGS= 2024-08-31T22:14:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-08-31T22:14:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:14:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ CEPH_ARGS= 2024-08-31T22:14:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror pool 2024-08-31T22:14:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-08-31T22:14:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-08-31T22:14:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-08-31T22:14:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 pool 2024-08-31T22:14:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-08-31T22:14:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -z ']' 2024-08-31T22:14:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -z ']' 2024-08-31T22:14:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ peer_add cluster2 mirror cluster1 2024-08-31T22:14:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:14:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:14:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local client_cluster=cluster1 2024-08-31T22:14:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local remote_cluster=cluster1 2024-08-31T22:14:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local uuid_var_name 2024-08-31T22:14:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:14:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local error_code 2024-08-31T22:14:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local peer_uuid 2024-08-31T22:14:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 16 32 2024-08-31T22:14:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:14:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-08-31T22:14:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ peer_uuid=1fe06a2f-0ee0-4bef-bae3-4e7556295e87 2024-08-31T22:14:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=0 2024-08-31T22:14:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:14:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 -eq 17 ']' 2024-08-31T22:14:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 0 -eq 0 2024-08-31T22:14:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:14:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:14:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-08-31T22:14:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:14:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror_parent 2024-08-31T22:14:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local client_cluster=cluster1 2024-08-31T22:14:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local remote_cluster=cluster1 2024-08-31T22:14:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local uuid_var_name 2024-08-31T22:14:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:14:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local error_code 2024-08-31T22:14:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local peer_uuid 2024-08-31T22:14:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 16 32 2024-08-31T22:14:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:14:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-08-31T22:14:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ peer_uuid=1e2352e4-0380-4407-9a5e-5dd9dae1a55c 2024-08-31T22:14:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=0 2024-08-31T22:14:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:14:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 -eq 17 ']' 2024-08-31T22:14:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 0 -eq 0 2024-08-31T22:14:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:14:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:14:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n octopus ']' 2024-08-31T22:14:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ CEPH_ARGS= 2024-08-31T22:14:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --cluster cluster1 osd set-require-min-compat-client octopus 2024-08-31T22:14:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:set require_min_compat_client to octopus 2024-08-31T22:14:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ CEPH_ARGS= 2024-08-31T22:14:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --cluster cluster2 osd set-require-min-compat-client octopus 2024-08-31T22:14:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:set require_min_compat_client to octopus 2024-08-31T22:14:55.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ start_mirrors cluster1 2024-08-31T22:14:55.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:14:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:14:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:14:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ start_mirror cluster1:0 2024-08-31T22:14:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:14:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:14:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:14:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:14:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:14:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:14:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:14:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:14:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:14:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:14:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:14:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:14:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:14:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:14:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-08-31T22:14:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:14:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ start_mirror cluster1:1 2024-08-31T22:14:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:14:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:14:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:14:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:14:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:14:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:14:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:14:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:14:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:14:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:14:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:14:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:14:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:14:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:14:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-08-31T22:14:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ start_mirrors cluster2 2024-08-31T22:14:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:14:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:14:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:14:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ start_mirror cluster2:0 2024-08-31T22:14:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2:0 2024-08-31T22:14:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:14:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-08-31T22:14:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster2:0 2024-08-31T22:14:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:14:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:14:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster2 2024-08-31T22:14:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:14:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster2:0 ']' 2024-08-31T22:14:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster2 2024-08-31T22:14:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster2 2024-08-31T22:14:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:14:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:14:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:14:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-08-31T22:14:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:14:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ start_mirror cluster2:1 2024-08-31T22:14:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2:1 2024-08-31T22:14:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:14:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-08-31T22:14:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster2:1 2024-08-31T22:14:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:14:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:14:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster2 2024-08-31T22:14:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:14:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster2:1 ']' 2024-08-31T22:14:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster2 2024-08-31T22:14:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster2 2024-08-31T22:14:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:14:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:14:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:14:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd-mirror --cluster cluster2 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-08-31T22:14:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-08-31T22:14:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ tee -a /tmp/tmp.iYfzsjLopX/rbd-mirror.test.log 2024-08-31T22:14:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ date '+%F %T' 2024-08-31T22:14:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo 2024-08-31 22:14:55 TEST: add image and test replay after client crashes 2024-08-31T22:14:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:2024-08-31 22:14:55 TEST: add image and test replay after client crashes 2024-08-31T22:14:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=test 2024-08-31T22:14:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror test journal 512M 2024-08-31T22:14:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:14:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:14:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:14:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:14:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:14:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror test 512M 2024-08-31T22:14:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:14:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:14:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:14:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:14:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 512M ']' 2024-08-31T22:14:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=512M 2024-08-31T22:14:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:14:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/test 2024-08-31T22:14:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:14:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:14:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:14:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:14:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:14:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-08-31T22:14:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:14:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:14:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:14:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-08-31T22:14:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:14:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:14:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:14:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:14:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:14:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:14:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:14:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror test started 2024-08-31T22:14:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:14:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:14:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:14:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:14:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:14:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:14:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-08-31T22:14:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:14:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:14:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:14:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:14:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:14:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:14:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:14:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:14:56.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:14:56.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:14:56.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:14:56.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-08-31T22:14:56.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:14:56.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:14:56.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:14:56.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:14:56.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:14:56.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:14:56.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:14:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:14:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:14:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:14:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:14:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:14:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:14:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:14:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:14:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:14:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:14:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:14:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:14:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:14:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:14:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:14:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:14:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:14:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:14:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:14:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:14:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:14:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:14:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:14:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:14:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:14:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:14:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-08-31T22:14:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-08-31T22:14:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:14:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:14:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:14:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-08-31T22:14:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:14:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:14:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:14:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:14:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:14:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:14:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:14:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:14:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:14:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:14:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:14:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:14:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:14:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:14:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:14:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:14:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:14:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:14:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:14:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:14:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:14:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:14:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:14:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:14:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:14:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:14:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:14:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:14:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:14:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:14:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:14:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:14:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:14:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-08-31T22:14:56.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-08-31T22:14:56.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:14:56.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:14:56.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:14:56.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 1 2024-08-31T22:14:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:14:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:14:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:14:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-08-31T22:14:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:14:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:14:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:14:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:14:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:14:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:14:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:14:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:14:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:14:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:14:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:14:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:14:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:14:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:14:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:14:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:14:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:14:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:14:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:14:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:14:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:14:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:14:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:14:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:14:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:14:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:14:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:14:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:14:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:14:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:14:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:14:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:14:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:14:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-08-31T22:14:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-08-31T22:14:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:14:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:14:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:14:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-08-31T22:14:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:14:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:14:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:14:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:14:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:14:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:14:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:14:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:14:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:14:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:14:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:14:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:14:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:14:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:14:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:14:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:14:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:14:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:14:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:14:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:14:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:14:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:14:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:14:57.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:14:57.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:14:57.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:14:57.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:14:57.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:14:57.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:14:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:14:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:14:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:14:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:14:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-08-31T22:14:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-08-31T22:14:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:14:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:14:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:14:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 2 2024-08-31T22:14:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:14:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:14:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:14:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-08-31T22:14:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:14:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:14:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:14:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:14:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:14:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:14:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:14:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:14:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:14:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:14:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:14:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:14:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:14:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:14:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:14:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:14:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:14:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:14:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:14:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:14:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:14:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:14:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:14:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:14:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:14:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:14:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:14:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:14:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:14:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:14:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:14:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:14:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:14:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-08-31T22:14:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:14:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:14:59.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:14:59.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:14:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:14:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:14:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:14:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:14:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ clean_snap_name= 2024-08-31T22:14:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 1 10 2024-08-31T22:14:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:14:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror test 2024-08-31T22:14:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:14:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:14:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:14:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:14:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=12 2024-08-31T22:14:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:14:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-08-31T22:15:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 85576704 bytes 2024-08-31T22:15:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 121300992 bytes 2024-08-31T22:15:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 174966784 bytes 2024-08-31T22:15:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 222706688 bytes 2024-08-31T22:15:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:500 IOs, 326288384 bytes 2024-08-31T22:15:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:15:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:15:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:15:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-08-31T22:15:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:15:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:15:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:15:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:12.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:15:13.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-08-31T22:15:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:15:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:15:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:15:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:15:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:15:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-08-31T22:15:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='test: 2024-08-31T22:15:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: global_id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:15:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+replaying 2024-08-31T22:15:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-08-31T22:15:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: service: mirror.0 on smithi032 2024-08-31T22:15:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:14:59 2024-08-31T22:15:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: peer_sites: 2024-08-31T22:15:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: name: cluster2 2024-08-31T22:15:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+stopped 2024-08-31T22:15:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: local image is primary 2024-08-31T22:15:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:15:10' 2024-08-31T22:15:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ state: .*up+replaying' 2024-08-31T22:15:13.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: state: up+replaying 2024-08-31T22:15:13.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ description: .*' 2024-08-31T22:15:13.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-08-31T22:15:13.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:15:13.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:15:13.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^up+' 2024-08-31T22:15:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:15:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep 'service: *mirror..* on ' 2024-08-31T22:15:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: service: mirror.0 on smithi032 2024-08-31T22:15:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update 2024-08-31T22:15:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-08-31T22:15:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:14:59' 2024-08-31T22:15:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-08-31 22:14:59' 2024-08-31T22:15:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:15:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'prev_last_update=2024-08-31 22:14:59' 2024-08-31T22:15:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:15:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:15:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-08-31T22:15:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='OKOKOK142574264mirror.0smithi03219.1.1-244-g3091dfc0trueOK42614267mirror.1smithi03219.1.1-244-g3091dfc0falseOKtestdd2c705e-25cf-4565-981f-d48de48eee84up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}42574264mirror.0smithi0322024-08-31 22:14:59cluster24ca32b00-2164-486e-837e-e93795a16af1up+stoppedlocal image is primary2024-08-31 22:15:10' 2024-08-31T22:15:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update state 2024-08-31T22:15:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-08-31T22:15:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:14:59' 2024-08-31T22:15:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-08-31T22:15:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ state=up+replaying 2024-08-31T22:15:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:15:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep up+replaying 2024-08-31T22:15:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:15:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap1 2024-08-31T22:15:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap1 2024-08-31T22:15:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:15:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap1 2024-08-31T22:15:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:15:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-08-31T22:15:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-08-31T22:15:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:15:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:15:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:15:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror test started 2024-08-31T22:15:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:15:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:15:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:15:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-08-31T22:15:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:15:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:15:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:15:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:15:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:15:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:15:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:15:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-08-31T22:15:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:15:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:15:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:15:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:15:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:15:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:15:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:15:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:15:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:15:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:15:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:15:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:15:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:15:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:15:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:15:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:15:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-08-31T22:15:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:15:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:15:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:15:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:15:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:15:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:15:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:15:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:15:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:15:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:15:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:15:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:15:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:15:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:15:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:15:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:15:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:44.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:44.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:44.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:44.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:44.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:15:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=10257], [object_number=12, tag_tid=2, entry_tid=10256], [object_number=11, tag_tid=2, entry_tid=10255], [object_number=10, tag_tid=2, entry_tid=10254]]], state=connected] 2024-08-31T22:15:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:44.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:15:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=10257], [object_number=12, tag_tid=2, entry_tid=10256], [object_number=11, tag_tid=2, entry_tid=10255], [object_number=10, tag_tid=2, entry_tid=10254]]], state=connected] 2024-08-31T22:15:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=10257' 2024-08-31T22:15:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=10257' 2024-08-31T22:15:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=10257' '!=' '' 2024-08-31T22:15:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=10257' = '' 2024-08-31T22:15:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=10257' 2024-08-31T22:15:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=10257' 2024-08-31T22:15:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=10257' 2024-08-31T22:15:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10257 2024-08-31T22:15:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 10257 2024-08-31T22:15:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=10355], [object_number=14, tag_tid=2, entry_tid=10354], [object_number=13, tag_tid=2, entry_tid=10353], [object_number=12, tag_tid=2, entry_tid=10352]]], state=connected] 2024-08-31T22:15:45.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=10355], [object_number=14, tag_tid=2, entry_tid=10354], [object_number=13, tag_tid=2, entry_tid=10353], [object_number=12, tag_tid=2, entry_tid=10352]]], state=connected] 2024-08-31T22:15:45.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=10355' 2024-08-31T22:15:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=10355' 2024-08-31T22:15:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=10355' '!=' 'object_number=13, tag_tid=2, entry_tid=10257' 2024-08-31T22:15:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=10355' = 'object_number=13, tag_tid=2, entry_tid=10257' 2024-08-31T22:15:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=10355' 2024-08-31T22:15:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=10355' 2024-08-31T22:15:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=10355' 2024-08-31T22:15:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10355 2024-08-31T22:15:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 10355 2024-08-31T22:15:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:45.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:45.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=10485], [object_number=12, tag_tid=2, entry_tid=10484], [object_number=15, tag_tid=2, entry_tid=10483], [object_number=14, tag_tid=2, entry_tid=10482]]], state=connected] 2024-08-31T22:15:45.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:45.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=10485], [object_number=12, tag_tid=2, entry_tid=10484], [object_number=15, tag_tid=2, entry_tid=10483], [object_number=14, tag_tid=2, entry_tid=10482]]], state=connected] 2024-08-31T22:15:45.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=10485' 2024-08-31T22:15:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=10485' 2024-08-31T22:15:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=10485' '!=' 'object_number=15, tag_tid=2, entry_tid=10355' 2024-08-31T22:15:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=10485' = 'object_number=15, tag_tid=2, entry_tid=10355' 2024-08-31T22:15:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=10485' 2024-08-31T22:15:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=10485' 2024-08-31T22:15:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:45.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:45.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:45.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:45.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=10485' 2024-08-31T22:15:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10485 2024-08-31T22:15:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 10485 2024-08-31T22:15:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:45.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:45.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:45.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:45.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:45.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:45.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=10606], [object_number=13, tag_tid=2, entry_tid=10605], [object_number=12, tag_tid=2, entry_tid=10604], [object_number=15, tag_tid=2, entry_tid=10603]]], state=connected] 2024-08-31T22:15:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=10606], [object_number=13, tag_tid=2, entry_tid=10605], [object_number=12, tag_tid=2, entry_tid=10604], [object_number=15, tag_tid=2, entry_tid=10603]]], state=connected] 2024-08-31T22:15:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:46.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=10606' 2024-08-31T22:15:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=10606' 2024-08-31T22:15:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=10606' '!=' 'object_number=13, tag_tid=2, entry_tid=10485' 2024-08-31T22:15:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=10606' = 'object_number=13, tag_tid=2, entry_tid=10485' 2024-08-31T22:15:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=10606' 2024-08-31T22:15:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:46.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=10606' 2024-08-31T22:15:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=10606' 2024-08-31T22:15:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10606 2024-08-31T22:15:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 10606 2024-08-31T22:15:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:46.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=10725], [object_number=12, tag_tid=2, entry_tid=10724], [object_number=15, tag_tid=2, entry_tid=10723], [object_number=14, tag_tid=2, entry_tid=10722]]], state=connected] 2024-08-31T22:15:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:46.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:46.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:46.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:46.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:46.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=10725], [object_number=12, tag_tid=2, entry_tid=10724], [object_number=15, tag_tid=2, entry_tid=10723], [object_number=14, tag_tid=2, entry_tid=10722]]], state=connected] 2024-08-31T22:15:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:46.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=10725' 2024-08-31T22:15:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=10725' 2024-08-31T22:15:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=10725' '!=' 'object_number=14, tag_tid=2, entry_tid=10606' 2024-08-31T22:15:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=10725' = 'object_number=14, tag_tid=2, entry_tid=10606' 2024-08-31T22:15:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=10725' 2024-08-31T22:15:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=10725' 2024-08-31T22:15:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=10725' 2024-08-31T22:15:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10725 2024-08-31T22:15:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 10725 2024-08-31T22:15:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:47.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=10847], [object_number=14, tag_tid=2, entry_tid=10846], [object_number=13, tag_tid=2, entry_tid=10845], [object_number=12, tag_tid=2, entry_tid=10844]]], state=connected] 2024-08-31T22:15:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=10847], [object_number=14, tag_tid=2, entry_tid=10846], [object_number=13, tag_tid=2, entry_tid=10845], [object_number=12, tag_tid=2, entry_tid=10844]]], state=connected] 2024-08-31T22:15:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=10847' 2024-08-31T22:15:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=10847' 2024-08-31T22:15:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=10847' '!=' 'object_number=13, tag_tid=2, entry_tid=10725' 2024-08-31T22:15:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=10847' = 'object_number=13, tag_tid=2, entry_tid=10725' 2024-08-31T22:15:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=10847' 2024-08-31T22:15:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=10847' 2024-08-31T22:15:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:47.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:47.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=10847' 2024-08-31T22:15:47.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:47.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10847 2024-08-31T22:15:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 10847 2024-08-31T22:15:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=10963], [object_number=14, tag_tid=2, entry_tid=10962], [object_number=13, tag_tid=2, entry_tid=10961], [object_number=12, tag_tid=2, entry_tid=10960]]], state=connected] 2024-08-31T22:15:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=10963], [object_number=14, tag_tid=2, entry_tid=10962], [object_number=13, tag_tid=2, entry_tid=10961], [object_number=12, tag_tid=2, entry_tid=10960]]], state=connected] 2024-08-31T22:15:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=10963' 2024-08-31T22:15:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=10963' 2024-08-31T22:15:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=10963' '!=' 'object_number=15, tag_tid=2, entry_tid=10847' 2024-08-31T22:15:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=10963' = 'object_number=15, tag_tid=2, entry_tid=10847' 2024-08-31T22:15:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=10963' 2024-08-31T22:15:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=10963' 2024-08-31T22:15:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=10963' 2024-08-31T22:15:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10963 2024-08-31T22:15:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 10963 2024-08-31T22:15:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:48.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:48.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:48.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=11077], [object_number=12, tag_tid=2, entry_tid=11076], [object_number=15, tag_tid=2, entry_tid=11075], [object_number=14, tag_tid=2, entry_tid=11074]]], state=connected] 2024-08-31T22:15:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=11077], [object_number=12, tag_tid=2, entry_tid=11076], [object_number=15, tag_tid=2, entry_tid=11075], [object_number=14, tag_tid=2, entry_tid=11074]]], state=connected] 2024-08-31T22:15:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=11077' 2024-08-31T22:15:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=11077' 2024-08-31T22:15:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=11077' '!=' 'object_number=15, tag_tid=2, entry_tid=10963' 2024-08-31T22:15:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=11077' = 'object_number=15, tag_tid=2, entry_tid=10963' 2024-08-31T22:15:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=11077' 2024-08-31T22:15:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=11077' 2024-08-31T22:15:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:48.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:48.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=11077' 2024-08-31T22:15:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11077 2024-08-31T22:15:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 11077 2024-08-31T22:15:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=11246], [object_number=13, tag_tid=2, entry_tid=11245], [object_number=12, tag_tid=2, entry_tid=11244], [object_number=15, tag_tid=2, entry_tid=11243]]], state=connected] 2024-08-31T22:15:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=11246], [object_number=13, tag_tid=2, entry_tid=11245], [object_number=12, tag_tid=2, entry_tid=11244], [object_number=15, tag_tid=2, entry_tid=11243]]], state=connected] 2024-08-31T22:15:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=11246' 2024-08-31T22:15:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=11246' 2024-08-31T22:15:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=11246' '!=' 'object_number=13, tag_tid=2, entry_tid=11077' 2024-08-31T22:15:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=11246' = 'object_number=13, tag_tid=2, entry_tid=11077' 2024-08-31T22:15:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=11246' 2024-08-31T22:15:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=11246' 2024-08-31T22:15:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=11246' 2024-08-31T22:15:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11246 2024-08-31T22:15:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 11246 2024-08-31T22:15:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:49.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:49.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:49.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:49.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:49.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:49.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=11362], [object_number=13, tag_tid=2, entry_tid=11361], [object_number=12, tag_tid=2, entry_tid=11360], [object_number=15, tag_tid=2, entry_tid=11359]]], state=connected] 2024-08-31T22:15:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=11362], [object_number=13, tag_tid=2, entry_tid=11361], [object_number=12, tag_tid=2, entry_tid=11360], [object_number=15, tag_tid=2, entry_tid=11359]]], state=connected] 2024-08-31T22:15:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=11362' 2024-08-31T22:15:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=11362' 2024-08-31T22:15:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=11362' '!=' 'object_number=14, tag_tid=2, entry_tid=11246' 2024-08-31T22:15:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=11362' = 'object_number=14, tag_tid=2, entry_tid=11246' 2024-08-31T22:15:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=11362' 2024-08-31T22:15:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=11362' 2024-08-31T22:15:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:49.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:49.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:49.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=11362' 2024-08-31T22:15:49.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:49.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11362 2024-08-31T22:15:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 11362 2024-08-31T22:15:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=11485], [object_number=12, tag_tid=2, entry_tid=11484], [object_number=15, tag_tid=2, entry_tid=11483], [object_number=14, tag_tid=2, entry_tid=11482]]], state=connected] 2024-08-31T22:15:49.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=11485], [object_number=12, tag_tid=2, entry_tid=11484], [object_number=15, tag_tid=2, entry_tid=11483], [object_number=14, tag_tid=2, entry_tid=11482]]], state=connected] 2024-08-31T22:15:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=11485' 2024-08-31T22:15:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=11485' 2024-08-31T22:15:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=11485' '!=' 'object_number=14, tag_tid=2, entry_tid=11362' 2024-08-31T22:15:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=11485' = 'object_number=14, tag_tid=2, entry_tid=11362' 2024-08-31T22:15:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=11485' 2024-08-31T22:15:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=11485' 2024-08-31T22:15:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=11485' 2024-08-31T22:15:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11485 2024-08-31T22:15:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 11485 2024-08-31T22:15:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:50.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=11604], [object_number=15, tag_tid=2, entry_tid=11603], [object_number=14, tag_tid=2, entry_tid=11602], [object_number=13, tag_tid=2, entry_tid=11601]]], state=connected] 2024-08-31T22:15:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=11604], [object_number=15, tag_tid=2, entry_tid=11603], [object_number=14, tag_tid=2, entry_tid=11602], [object_number=13, tag_tid=2, entry_tid=11601]]], state=connected] 2024-08-31T22:15:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=11604' 2024-08-31T22:15:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=11604' 2024-08-31T22:15:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=11604' '!=' 'object_number=13, tag_tid=2, entry_tid=11485' 2024-08-31T22:15:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=11604' = 'object_number=13, tag_tid=2, entry_tid=11485' 2024-08-31T22:15:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=11604' 2024-08-31T22:15:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=11604' 2024-08-31T22:15:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=11604' 2024-08-31T22:15:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11604 2024-08-31T22:15:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 11604 2024-08-31T22:15:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=11728], [object_number=15, tag_tid=2, entry_tid=11727], [object_number=14, tag_tid=2, entry_tid=11726], [object_number=13, tag_tid=2, entry_tid=11725]]], state=connected] 2024-08-31T22:15:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=11728], [object_number=15, tag_tid=2, entry_tid=11727], [object_number=14, tag_tid=2, entry_tid=11726], [object_number=13, tag_tid=2, entry_tid=11725]]], state=connected] 2024-08-31T22:15:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:50.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=11728' 2024-08-31T22:15:50.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=11728' 2024-08-31T22:15:50.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=11728' '!=' 'object_number=12, tag_tid=2, entry_tid=11604' 2024-08-31T22:15:50.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:50.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=11728' = 'object_number=12, tag_tid=2, entry_tid=11604' 2024-08-31T22:15:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=11728' 2024-08-31T22:15:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=11728' 2024-08-31T22:15:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=11728' 2024-08-31T22:15:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11728 2024-08-31T22:15:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 11728 2024-08-31T22:15:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:51.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:51.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:51.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:51.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:51.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:51.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=11866], [object_number=13, tag_tid=2, entry_tid=11865], [object_number=12, tag_tid=2, entry_tid=11864], [object_number=15, tag_tid=2, entry_tid=11863]]], state=connected] 2024-08-31T22:15:51.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=11866], [object_number=13, tag_tid=2, entry_tid=11865], [object_number=12, tag_tid=2, entry_tid=11864], [object_number=15, tag_tid=2, entry_tid=11863]]], state=connected] 2024-08-31T22:15:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=11866' 2024-08-31T22:15:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=11866' 2024-08-31T22:15:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=11866' '!=' 'object_number=12, tag_tid=2, entry_tid=11728' 2024-08-31T22:15:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=11866' = 'object_number=12, tag_tid=2, entry_tid=11728' 2024-08-31T22:15:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=11866' 2024-08-31T22:15:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=11866' 2024-08-31T22:15:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=11866' 2024-08-31T22:15:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11866 2024-08-31T22:15:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 11866 2024-08-31T22:15:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=11981], [object_number=12, tag_tid=2, entry_tid=11980], [object_number=15, tag_tid=2, entry_tid=11979], [object_number=14, tag_tid=2, entry_tid=11978]]], state=connected] 2024-08-31T22:15:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:51.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:51.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:51.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:51.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:51.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:51.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:51.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=11981], [object_number=12, tag_tid=2, entry_tid=11980], [object_number=15, tag_tid=2, entry_tid=11979], [object_number=14, tag_tid=2, entry_tid=11978]]], state=connected] 2024-08-31T22:15:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=11981' 2024-08-31T22:15:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=11981' 2024-08-31T22:15:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=11981' '!=' 'object_number=14, tag_tid=2, entry_tid=11866' 2024-08-31T22:15:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=11981' = 'object_number=14, tag_tid=2, entry_tid=11866' 2024-08-31T22:15:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=11981' 2024-08-31T22:15:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=11981' 2024-08-31T22:15:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=11981' 2024-08-31T22:15:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11981 2024-08-31T22:15:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 11981 2024-08-31T22:15:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:52.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:52.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:52.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:52.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:52.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12102], [object_number=13, tag_tid=2, entry_tid=12101], [object_number=12, tag_tid=2, entry_tid=12100], [object_number=15, tag_tid=2, entry_tid=12099]]], state=connected] 2024-08-31T22:15:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12102], [object_number=13, tag_tid=2, entry_tid=12101], [object_number=12, tag_tid=2, entry_tid=12100], [object_number=15, tag_tid=2, entry_tid=12099]]], state=connected] 2024-08-31T22:15:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=12102' 2024-08-31T22:15:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=12102' 2024-08-31T22:15:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12102' '!=' 'object_number=13, tag_tid=2, entry_tid=11981' 2024-08-31T22:15:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12102' = 'object_number=13, tag_tid=2, entry_tid=11981' 2024-08-31T22:15:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=12102' 2024-08-31T22:15:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12102' 2024-08-31T22:15:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12102' 2024-08-31T22:15:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12102 2024-08-31T22:15:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 12102 2024-08-31T22:15:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=12221], [object_number=12, tag_tid=2, entry_tid=12220], [object_number=15, tag_tid=2, entry_tid=12219], [object_number=14, tag_tid=2, entry_tid=12218]]], state=connected] 2024-08-31T22:15:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=12221], [object_number=12, tag_tid=2, entry_tid=12220], [object_number=15, tag_tid=2, entry_tid=12219], [object_number=14, tag_tid=2, entry_tid=12218]]], state=connected] 2024-08-31T22:15:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:52.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=12221' 2024-08-31T22:15:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=12221' 2024-08-31T22:15:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=12221' '!=' 'object_number=14, tag_tid=2, entry_tid=12102' 2024-08-31T22:15:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=12221' = 'object_number=14, tag_tid=2, entry_tid=12102' 2024-08-31T22:15:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=12221' 2024-08-31T22:15:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=12221' 2024-08-31T22:15:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:52.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:52.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=12221' 2024-08-31T22:15:52.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:52.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12221 2024-08-31T22:15:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 12221 2024-08-31T22:15:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:53.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12342], [object_number=13, tag_tid=2, entry_tid=12341], [object_number=12, tag_tid=2, entry_tid=12340], [object_number=15, tag_tid=2, entry_tid=12339]]], state=connected] 2024-08-31T22:15:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12342], [object_number=13, tag_tid=2, entry_tid=12341], [object_number=12, tag_tid=2, entry_tid=12340], [object_number=15, tag_tid=2, entry_tid=12339]]], state=connected] 2024-08-31T22:15:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=12342' 2024-08-31T22:15:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=12342' 2024-08-31T22:15:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12342' '!=' 'object_number=13, tag_tid=2, entry_tid=12221' 2024-08-31T22:15:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12342' = 'object_number=13, tag_tid=2, entry_tid=12221' 2024-08-31T22:15:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=12342' 2024-08-31T22:15:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12342' 2024-08-31T22:15:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12342' 2024-08-31T22:15:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12342 2024-08-31T22:15:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 12342 2024-08-31T22:15:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:53.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:53.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:53.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:53.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:53.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=12463], [object_number=14, tag_tid=2, entry_tid=12462], [object_number=13, tag_tid=2, entry_tid=12461], [object_number=12, tag_tid=2, entry_tid=12460]]], state=connected] 2024-08-31T22:15:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=12463], [object_number=14, tag_tid=2, entry_tid=12462], [object_number=13, tag_tid=2, entry_tid=12461], [object_number=12, tag_tid=2, entry_tid=12460]]], state=connected] 2024-08-31T22:15:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=12463' 2024-08-31T22:15:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=12463' 2024-08-31T22:15:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=12463' '!=' 'object_number=14, tag_tid=2, entry_tid=12342' 2024-08-31T22:15:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=12463' = 'object_number=14, tag_tid=2, entry_tid=12342' 2024-08-31T22:15:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=12463' 2024-08-31T22:15:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=12463' 2024-08-31T22:15:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=12463' 2024-08-31T22:15:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12463 2024-08-31T22:15:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 12463 2024-08-31T22:15:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12582], [object_number=13, tag_tid=2, entry_tid=12581], [object_number=12, tag_tid=2, entry_tid=12580], [object_number=15, tag_tid=2, entry_tid=12579]]], state=connected] 2024-08-31T22:15:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:54.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12582], [object_number=13, tag_tid=2, entry_tid=12581], [object_number=12, tag_tid=2, entry_tid=12580], [object_number=15, tag_tid=2, entry_tid=12579]]], state=connected] 2024-08-31T22:15:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=12582' 2024-08-31T22:15:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=12582' 2024-08-31T22:15:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12582' '!=' 'object_number=15, tag_tid=2, entry_tid=12463' 2024-08-31T22:15:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12582' = 'object_number=15, tag_tid=2, entry_tid=12463' 2024-08-31T22:15:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=12582' 2024-08-31T22:15:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12582' 2024-08-31T22:15:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12582' 2024-08-31T22:15:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12582 2024-08-31T22:15:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 12582 2024-08-31T22:15:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:54.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12698], [object_number=13, tag_tid=2, entry_tid=12697], [object_number=12, tag_tid=2, entry_tid=12696], [object_number=15, tag_tid=2, entry_tid=12695]]], state=connected] 2024-08-31T22:15:54.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:54.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:54.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:54.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:54.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:54.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:54.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12698], [object_number=13, tag_tid=2, entry_tid=12697], [object_number=12, tag_tid=2, entry_tid=12696], [object_number=15, tag_tid=2, entry_tid=12695]]], state=connected] 2024-08-31T22:15:54.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:54.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=12698' 2024-08-31T22:15:54.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=12698' 2024-08-31T22:15:54.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12698' '!=' 'object_number=14, tag_tid=2, entry_tid=12582' 2024-08-31T22:15:54.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:54.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12698' = 'object_number=14, tag_tid=2, entry_tid=12582' 2024-08-31T22:15:54.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=12698' 2024-08-31T22:15:54.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:54.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:54.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12698' 2024-08-31T22:15:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:54.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:54.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12698' 2024-08-31T22:15:54.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12698 2024-08-31T22:15:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 12698 2024-08-31T22:15:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:55.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:55.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=12824], [object_number=15, tag_tid=2, entry_tid=12823], [object_number=14, tag_tid=2, entry_tid=12822], [object_number=13, tag_tid=2, entry_tid=12821]]], state=connected] 2024-08-31T22:15:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=12824], [object_number=15, tag_tid=2, entry_tid=12823], [object_number=14, tag_tid=2, entry_tid=12822], [object_number=13, tag_tid=2, entry_tid=12821]]], state=connected] 2024-08-31T22:15:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=12824' 2024-08-31T22:15:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=12824' 2024-08-31T22:15:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=12824' '!=' 'object_number=14, tag_tid=2, entry_tid=12698' 2024-08-31T22:15:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=12824' = 'object_number=14, tag_tid=2, entry_tid=12698' 2024-08-31T22:15:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=12824' 2024-08-31T22:15:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=12824' 2024-08-31T22:15:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=12824' 2024-08-31T22:15:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12824 2024-08-31T22:15:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 12824 2024-08-31T22:15:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:55.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:55.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:55.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:55.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:55.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:55.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:55.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:55.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:55.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=12948], [object_number=15, tag_tid=2, entry_tid=12947], [object_number=14, tag_tid=2, entry_tid=12946], [object_number=13, tag_tid=2, entry_tid=12945]]], state=connected] 2024-08-31T22:15:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:55.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:55.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=12948], [object_number=15, tag_tid=2, entry_tid=12947], [object_number=14, tag_tid=2, entry_tid=12946], [object_number=13, tag_tid=2, entry_tid=12945]]], state=connected] 2024-08-31T22:15:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=12948' 2024-08-31T22:15:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=12948' 2024-08-31T22:15:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=12948' '!=' 'object_number=12, tag_tid=2, entry_tid=12824' 2024-08-31T22:15:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=12948' = 'object_number=12, tag_tid=2, entry_tid=12824' 2024-08-31T22:15:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=12948' 2024-08-31T22:15:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=12948' 2024-08-31T22:15:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:55.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:55.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:55.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=12948' 2024-08-31T22:15:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12948 2024-08-31T22:15:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 12948 2024-08-31T22:15:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:56.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:56.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:56.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=13052], [object_number=15, tag_tid=2, entry_tid=13051], [object_number=14, tag_tid=2, entry_tid=13050], [object_number=13, tag_tid=2, entry_tid=13049]]], state=connected] 2024-08-31T22:15:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:56.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=13052], [object_number=15, tag_tid=2, entry_tid=13051], [object_number=14, tag_tid=2, entry_tid=13050], [object_number=13, tag_tid=2, entry_tid=13049]]], state=connected] 2024-08-31T22:15:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=13052' 2024-08-31T22:15:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=13052' 2024-08-31T22:15:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=13052' '!=' 'object_number=12, tag_tid=2, entry_tid=12948' 2024-08-31T22:15:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=13052' = 'object_number=12, tag_tid=2, entry_tid=12948' 2024-08-31T22:15:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=13052' 2024-08-31T22:15:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=13052' 2024-08-31T22:15:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=13052' 2024-08-31T22:15:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13052 2024-08-31T22:15:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 13052 2024-08-31T22:15:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:57.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:57.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:57.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=13254], [object_number=13, tag_tid=2, entry_tid=13253], [object_number=12, tag_tid=2, entry_tid=13252], [object_number=15, tag_tid=2, entry_tid=13251]]], state=connected] 2024-08-31T22:15:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=13254], [object_number=13, tag_tid=2, entry_tid=13253], [object_number=12, tag_tid=2, entry_tid=13252], [object_number=15, tag_tid=2, entry_tid=13251]]], state=connected] 2024-08-31T22:15:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=13254' 2024-08-31T22:15:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=13254' 2024-08-31T22:15:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=13254' '!=' 'object_number=12, tag_tid=2, entry_tid=13052' 2024-08-31T22:15:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=13254' = 'object_number=12, tag_tid=2, entry_tid=13052' 2024-08-31T22:15:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=13254' 2024-08-31T22:15:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=13254' 2024-08-31T22:15:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=13254' 2024-08-31T22:15:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13254 2024-08-31T22:15:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 13254 2024-08-31T22:15:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13359], [object_number=14, tag_tid=2, entry_tid=13358], [object_number=13, tag_tid=2, entry_tid=13357], [object_number=12, tag_tid=2, entry_tid=13356]]], state=connected] 2024-08-31T22:15:57.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13359], [object_number=14, tag_tid=2, entry_tid=13358], [object_number=13, tag_tid=2, entry_tid=13357], [object_number=12, tag_tid=2, entry_tid=13356]]], state=connected] 2024-08-31T22:15:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=13359' 2024-08-31T22:15:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=13359' 2024-08-31T22:15:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13359' '!=' 'object_number=14, tag_tid=2, entry_tid=13254' 2024-08-31T22:15:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13359' = 'object_number=14, tag_tid=2, entry_tid=13254' 2024-08-31T22:15:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=13359' 2024-08-31T22:15:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13359' 2024-08-31T22:15:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13359' 2024-08-31T22:15:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13359 2024-08-31T22:15:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 13359 2024-08-31T22:15:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:57.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:58.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:58.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:58.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=13452], [object_number=15, tag_tid=2, entry_tid=13451], [object_number=14, tag_tid=2, entry_tid=13450], [object_number=13, tag_tid=2, entry_tid=13449]]], state=connected] 2024-08-31T22:15:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:58.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:58.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:58.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:58.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:58.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=13452], [object_number=15, tag_tid=2, entry_tid=13451], [object_number=14, tag_tid=2, entry_tid=13450], [object_number=13, tag_tid=2, entry_tid=13449]]], state=connected] 2024-08-31T22:15:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=13452' 2024-08-31T22:15:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=13452' 2024-08-31T22:15:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=13452' '!=' 'object_number=15, tag_tid=2, entry_tid=13359' 2024-08-31T22:15:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=13452' = 'object_number=15, tag_tid=2, entry_tid=13359' 2024-08-31T22:15:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=13452' 2024-08-31T22:15:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=13452' 2024-08-31T22:15:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:58.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=13452' 2024-08-31T22:15:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13452 2024-08-31T22:15:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 13452 2024-08-31T22:15:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:58.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=13557], [object_number=12, tag_tid=2, entry_tid=13556], [object_number=15, tag_tid=2, entry_tid=13555], [object_number=14, tag_tid=2, entry_tid=13554]]], state=connected] 2024-08-31T22:15:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=13557], [object_number=12, tag_tid=2, entry_tid=13556], [object_number=15, tag_tid=2, entry_tid=13555], [object_number=14, tag_tid=2, entry_tid=13554]]], state=connected] 2024-08-31T22:15:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=13557' 2024-08-31T22:15:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=13557' 2024-08-31T22:15:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=13557' '!=' 'object_number=12, tag_tid=2, entry_tid=13452' 2024-08-31T22:15:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=13557' = 'object_number=12, tag_tid=2, entry_tid=13452' 2024-08-31T22:15:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=13557' 2024-08-31T22:15:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=13557' 2024-08-31T22:15:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:58.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=13557' 2024-08-31T22:15:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13557 2024-08-31T22:15:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 13557 2024-08-31T22:15:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:59.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:59.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:59.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=13661], [object_number=12, tag_tid=2, entry_tid=13660], [object_number=15, tag_tid=2, entry_tid=13659], [object_number=14, tag_tid=2, entry_tid=13658]]], state=connected] 2024-08-31T22:15:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:59.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=13661], [object_number=12, tag_tid=2, entry_tid=13660], [object_number=15, tag_tid=2, entry_tid=13659], [object_number=14, tag_tid=2, entry_tid=13658]]], state=connected] 2024-08-31T22:15:59.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=13661' 2024-08-31T22:15:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=13661' 2024-08-31T22:15:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=13661' '!=' 'object_number=13, tag_tid=2, entry_tid=13557' 2024-08-31T22:15:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=13661' = 'object_number=13, tag_tid=2, entry_tid=13557' 2024-08-31T22:15:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=13661' 2024-08-31T22:15:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=13661' 2024-08-31T22:15:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=13661' 2024-08-31T22:15:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13661 2024-08-31T22:15:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 13661 2024-08-31T22:15:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:15:59.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:15:59.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:15:59.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:59.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:59.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:59.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:15:59.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:59.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:59.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:59.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:15:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:59.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:59.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:59.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:59.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=13696], [object_number=19, tag_tid=2, entry_tid=13695], [object_number=18, tag_tid=2, entry_tid=13694], [object_number=13, tag_tid=2, entry_tid=13693]]], state=connected] 2024-08-31T22:15:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:15:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:15:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:15:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:15:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:15:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:15:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:15:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:15:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:15:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:15:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:15:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:15:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:15:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:15:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=13696], [object_number=19, tag_tid=2, entry_tid=13695], [object_number=18, tag_tid=2, entry_tid=13694], [object_number=13, tag_tid=2, entry_tid=13693]]], state=connected] 2024-08-31T22:15:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:15:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=13696' 2024-08-31T22:15:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=13696' 2024-08-31T22:15:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=13696' '!=' 'object_number=13, tag_tid=2, entry_tid=13661' 2024-08-31T22:15:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:15:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=13696' = 'object_number=13, tag_tid=2, entry_tid=13661' 2024-08-31T22:15:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=13696' 2024-08-31T22:15:59.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:59.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:59.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:59.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:15:59.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=13696' 2024-08-31T22:15:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:15:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:15:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:15:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:15:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=13696' 2024-08-31T22:15:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:15:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:15:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13696 2024-08-31T22:15:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 13696 2024-08-31T22:15:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:15:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:15:59.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:15:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:15:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:15:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:15:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:15:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:15:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:15:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:15:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:15:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:15:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:15:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:15:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:15:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:15:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:15:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:15:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:15:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:15:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:15:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:15:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:15:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:15:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:15:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:15:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:15:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:15:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:15:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:15:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:15:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:15:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:15:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:15:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:15:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:15:59.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:59.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:59.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:15:59.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:00.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:00.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=13798], [object_number=17, tag_tid=2, entry_tid=13797], [object_number=16, tag_tid=2, entry_tid=13796], [object_number=19, tag_tid=2, entry_tid=13795]]], state=connected] 2024-08-31T22:16:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:00.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:00.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:00.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:00.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=13798], [object_number=17, tag_tid=2, entry_tid=13797], [object_number=16, tag_tid=2, entry_tid=13796], [object_number=19, tag_tid=2, entry_tid=13795]]], state=connected] 2024-08-31T22:16:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=13798' 2024-08-31T22:16:00.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=13798' 2024-08-31T22:16:00.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=13798' '!=' 'object_number=16, tag_tid=2, entry_tid=13696' 2024-08-31T22:16:00.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:00.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=13798' = 'object_number=16, tag_tid=2, entry_tid=13696' 2024-08-31T22:16:00.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=13798' 2024-08-31T22:16:00.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=13798' 2024-08-31T22:16:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=13798' 2024-08-31T22:16:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13798 2024-08-31T22:16:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 13798 2024-08-31T22:16:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=13931], [object_number=18, tag_tid=2, entry_tid=13930], [object_number=17, tag_tid=2, entry_tid=13929], [object_number=16, tag_tid=2, entry_tid=13928]]], state=connected] 2024-08-31T22:16:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:00.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:00.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:00.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:00.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:00.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=13931], [object_number=18, tag_tid=2, entry_tid=13930], [object_number=17, tag_tid=2, entry_tid=13929], [object_number=16, tag_tid=2, entry_tid=13928]]], state=connected] 2024-08-31T22:16:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=13931' 2024-08-31T22:16:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=13931' 2024-08-31T22:16:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=13931' '!=' 'object_number=18, tag_tid=2, entry_tid=13798' 2024-08-31T22:16:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=13931' = 'object_number=18, tag_tid=2, entry_tid=13798' 2024-08-31T22:16:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=13931' 2024-08-31T22:16:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:00.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=13931' 2024-08-31T22:16:00.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:00.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:00.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=13931' 2024-08-31T22:16:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13931 2024-08-31T22:16:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 13931 2024-08-31T22:16:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:01.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=14051], [object_number=18, tag_tid=2, entry_tid=14050], [object_number=17, tag_tid=2, entry_tid=14049], [object_number=16, tag_tid=2, entry_tid=14048]]], state=connected] 2024-08-31T22:16:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=14051], [object_number=18, tag_tid=2, entry_tid=14050], [object_number=17, tag_tid=2, entry_tid=14049], [object_number=16, tag_tid=2, entry_tid=14048]]], state=connected] 2024-08-31T22:16:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=14051' 2024-08-31T22:16:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=14051' 2024-08-31T22:16:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=14051' '!=' 'object_number=19, tag_tid=2, entry_tid=13931' 2024-08-31T22:16:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=14051' = 'object_number=19, tag_tid=2, entry_tid=13931' 2024-08-31T22:16:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=14051' 2024-08-31T22:16:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:01.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=14051' 2024-08-31T22:16:01.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:01.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:01.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:01.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=14051' 2024-08-31T22:16:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14051 2024-08-31T22:16:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 14051 2024-08-31T22:16:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:01.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:01.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:01.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:01.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:01.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=14170], [object_number=17, tag_tid=2, entry_tid=14169], [object_number=16, tag_tid=2, entry_tid=14168], [object_number=19, tag_tid=2, entry_tid=14167]]], state=connected] 2024-08-31T22:16:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=14170], [object_number=17, tag_tid=2, entry_tid=14169], [object_number=16, tag_tid=2, entry_tid=14168], [object_number=19, tag_tid=2, entry_tid=14167]]], state=connected] 2024-08-31T22:16:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=14170' 2024-08-31T22:16:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=14170' 2024-08-31T22:16:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=14170' '!=' 'object_number=19, tag_tid=2, entry_tid=14051' 2024-08-31T22:16:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=14170' = 'object_number=19, tag_tid=2, entry_tid=14051' 2024-08-31T22:16:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=14170' 2024-08-31T22:16:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=14170' 2024-08-31T22:16:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=14170' 2024-08-31T22:16:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14170 2024-08-31T22:16:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 14170 2024-08-31T22:16:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:02.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:02.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:02.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=14277], [object_number=16, tag_tid=2, entry_tid=14276], [object_number=19, tag_tid=2, entry_tid=14275], [object_number=18, tag_tid=2, entry_tid=14274]]], state=connected] 2024-08-31T22:16:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=14277], [object_number=16, tag_tid=2, entry_tid=14276], [object_number=19, tag_tid=2, entry_tid=14275], [object_number=18, tag_tid=2, entry_tid=14274]]], state=connected] 2024-08-31T22:16:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=14277' 2024-08-31T22:16:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=14277' 2024-08-31T22:16:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=14277' '!=' 'object_number=18, tag_tid=2, entry_tid=14170' 2024-08-31T22:16:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=14277' = 'object_number=18, tag_tid=2, entry_tid=14170' 2024-08-31T22:16:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=17, tag_tid=2, entry_tid=14277' 2024-08-31T22:16:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=14277' 2024-08-31T22:16:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=14277' 2024-08-31T22:16:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14277 2024-08-31T22:16:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 14277 2024-08-31T22:16:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:02.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:02.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:02.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:02.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=14384], [object_number=19, tag_tid=2, entry_tid=14383], [object_number=18, tag_tid=2, entry_tid=14382], [object_number=17, tag_tid=2, entry_tid=14381]]], state=connected] 2024-08-31T22:16:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=14384], [object_number=19, tag_tid=2, entry_tid=14383], [object_number=18, tag_tid=2, entry_tid=14382], [object_number=17, tag_tid=2, entry_tid=14381]]], state=connected] 2024-08-31T22:16:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=14384' 2024-08-31T22:16:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=14384' 2024-08-31T22:16:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=14384' '!=' 'object_number=17, tag_tid=2, entry_tid=14277' 2024-08-31T22:16:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=14384' = 'object_number=17, tag_tid=2, entry_tid=14277' 2024-08-31T22:16:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=14384' 2024-08-31T22:16:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=14384' 2024-08-31T22:16:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=14384' 2024-08-31T22:16:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14384 2024-08-31T22:16:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 14384 2024-08-31T22:16:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:02.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:02.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:02.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:02.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:02.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:02.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:02.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:02.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:02.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:02.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:02.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:02.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:02.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=14491], [object_number=18, tag_tid=2, entry_tid=14490], [object_number=17, tag_tid=2, entry_tid=14489], [object_number=16, tag_tid=2, entry_tid=14488]]], state=connected] 2024-08-31T22:16:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=14491], [object_number=18, tag_tid=2, entry_tid=14490], [object_number=17, tag_tid=2, entry_tid=14489], [object_number=16, tag_tid=2, entry_tid=14488]]], state=connected] 2024-08-31T22:16:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=14491' 2024-08-31T22:16:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=14491' 2024-08-31T22:16:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=14491' '!=' 'object_number=16, tag_tid=2, entry_tid=14384' 2024-08-31T22:16:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=14491' = 'object_number=16, tag_tid=2, entry_tid=14384' 2024-08-31T22:16:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=14491' 2024-08-31T22:16:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=14491' 2024-08-31T22:16:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=14491' 2024-08-31T22:16:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14491 2024-08-31T22:16:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 14491 2024-08-31T22:16:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=14610], [object_number=17, tag_tid=2, entry_tid=14609], [object_number=16, tag_tid=2, entry_tid=14608], [object_number=19, tag_tid=2, entry_tid=14607]]], state=connected] 2024-08-31T22:16:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:03.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:03.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:03.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:03.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:03.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:03.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:03.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=14610], [object_number=17, tag_tid=2, entry_tid=14609], [object_number=16, tag_tid=2, entry_tid=14608], [object_number=19, tag_tid=2, entry_tid=14607]]], state=connected] 2024-08-31T22:16:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=14610' 2024-08-31T22:16:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=14610' 2024-08-31T22:16:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=14610' '!=' 'object_number=19, tag_tid=2, entry_tid=14491' 2024-08-31T22:16:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=14610' = 'object_number=19, tag_tid=2, entry_tid=14491' 2024-08-31T22:16:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=14610' 2024-08-31T22:16:03.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:03.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:03.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=14610' 2024-08-31T22:16:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=14610' 2024-08-31T22:16:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14610 2024-08-31T22:16:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 14610 2024-08-31T22:16:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:03.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:03.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:03.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:03.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:03.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=14730], [object_number=17, tag_tid=2, entry_tid=14729], [object_number=16, tag_tid=2, entry_tid=14728], [object_number=19, tag_tid=2, entry_tid=14727]]], state=connected] 2024-08-31T22:16:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:04.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:04.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:04.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=14730], [object_number=17, tag_tid=2, entry_tid=14729], [object_number=16, tag_tid=2, entry_tid=14728], [object_number=19, tag_tid=2, entry_tid=14727]]], state=connected] 2024-08-31T22:16:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=14730' 2024-08-31T22:16:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=14730' 2024-08-31T22:16:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=14730' '!=' 'object_number=18, tag_tid=2, entry_tid=14610' 2024-08-31T22:16:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=14730' = 'object_number=18, tag_tid=2, entry_tid=14610' 2024-08-31T22:16:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=14730' 2024-08-31T22:16:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=14730' 2024-08-31T22:16:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:04.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:04.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=14730' 2024-08-31T22:16:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14730 2024-08-31T22:16:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 14730 2024-08-31T22:16:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:04.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:04.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=14845], [object_number=16, tag_tid=2, entry_tid=14844], [object_number=19, tag_tid=2, entry_tid=14843], [object_number=18, tag_tid=2, entry_tid=14842]]], state=connected] 2024-08-31T22:16:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:04.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:04.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:04.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:04.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=14845], [object_number=16, tag_tid=2, entry_tid=14844], [object_number=19, tag_tid=2, entry_tid=14843], [object_number=18, tag_tid=2, entry_tid=14842]]], state=connected] 2024-08-31T22:16:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=14845' 2024-08-31T22:16:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=14845' 2024-08-31T22:16:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=14845' '!=' 'object_number=18, tag_tid=2, entry_tid=14730' 2024-08-31T22:16:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=14845' = 'object_number=18, tag_tid=2, entry_tid=14730' 2024-08-31T22:16:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=17, tag_tid=2, entry_tid=14845' 2024-08-31T22:16:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=14845' 2024-08-31T22:16:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=14845' 2024-08-31T22:16:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14845 2024-08-31T22:16:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 14845 2024-08-31T22:16:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=14972], [object_number=19, tag_tid=2, entry_tid=14971], [object_number=18, tag_tid=2, entry_tid=14970], [object_number=17, tag_tid=2, entry_tid=14969]]], state=connected] 2024-08-31T22:16:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=14972], [object_number=19, tag_tid=2, entry_tid=14971], [object_number=18, tag_tid=2, entry_tid=14970], [object_number=17, tag_tid=2, entry_tid=14969]]], state=connected] 2024-08-31T22:16:05.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=14972' 2024-08-31T22:16:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=14972' 2024-08-31T22:16:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=14972' '!=' 'object_number=17, tag_tid=2, entry_tid=14845' 2024-08-31T22:16:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=14972' = 'object_number=17, tag_tid=2, entry_tid=14845' 2024-08-31T22:16:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=14972' 2024-08-31T22:16:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=14972' 2024-08-31T22:16:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:05.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:05.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:05.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=14972' 2024-08-31T22:16:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14972 2024-08-31T22:16:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 14972 2024-08-31T22:16:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=15092], [object_number=19, tag_tid=2, entry_tid=15091], [object_number=18, tag_tid=2, entry_tid=15090], [object_number=17, tag_tid=2, entry_tid=15089]]], state=connected] 2024-08-31T22:16:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=15092], [object_number=19, tag_tid=2, entry_tid=15091], [object_number=18, tag_tid=2, entry_tid=15090], [object_number=17, tag_tid=2, entry_tid=15089]]], state=connected] 2024-08-31T22:16:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=15092' 2024-08-31T22:16:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=15092' 2024-08-31T22:16:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=15092' '!=' 'object_number=16, tag_tid=2, entry_tid=14972' 2024-08-31T22:16:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=15092' = 'object_number=16, tag_tid=2, entry_tid=14972' 2024-08-31T22:16:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=15092' 2024-08-31T22:16:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=15092' 2024-08-31T22:16:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=15092' 2024-08-31T22:16:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15092 2024-08-31T22:16:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 15092 2024-08-31T22:16:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=15223], [object_number=18, tag_tid=2, entry_tid=15222], [object_number=17, tag_tid=2, entry_tid=15221], [object_number=16, tag_tid=2, entry_tid=15220]]], state=connected] 2024-08-31T22:16:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=15223], [object_number=18, tag_tid=2, entry_tid=15222], [object_number=17, tag_tid=2, entry_tid=15221], [object_number=16, tag_tid=2, entry_tid=15220]]], state=connected] 2024-08-31T22:16:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=15223' 2024-08-31T22:16:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=15223' 2024-08-31T22:16:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=15223' '!=' 'object_number=16, tag_tid=2, entry_tid=15092' 2024-08-31T22:16:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=15223' = 'object_number=16, tag_tid=2, entry_tid=15092' 2024-08-31T22:16:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=15223' 2024-08-31T22:16:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=15223' 2024-08-31T22:16:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=15223' 2024-08-31T22:16:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15223 2024-08-31T22:16:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 15223 2024-08-31T22:16:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=15338], [object_number=17, tag_tid=2, entry_tid=15337], [object_number=16, tag_tid=2, entry_tid=15336], [object_number=19, tag_tid=2, entry_tid=15335]]], state=connected] 2024-08-31T22:16:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=15338], [object_number=17, tag_tid=2, entry_tid=15337], [object_number=16, tag_tid=2, entry_tid=15336], [object_number=19, tag_tid=2, entry_tid=15335]]], state=connected] 2024-08-31T22:16:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=15338' 2024-08-31T22:16:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=15338' 2024-08-31T22:16:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=15338' '!=' 'object_number=19, tag_tid=2, entry_tid=15223' 2024-08-31T22:16:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=15338' = 'object_number=19, tag_tid=2, entry_tid=15223' 2024-08-31T22:16:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=15338' 2024-08-31T22:16:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=15338' 2024-08-31T22:16:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=15338' 2024-08-31T22:16:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15338 2024-08-31T22:16:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 15338 2024-08-31T22:16:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:06.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=15451], [object_number=18, tag_tid=2, entry_tid=15450], [object_number=17, tag_tid=2, entry_tid=15449], [object_number=16, tag_tid=2, entry_tid=15448]]], state=connected] 2024-08-31T22:16:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=15451], [object_number=18, tag_tid=2, entry_tid=15450], [object_number=17, tag_tid=2, entry_tid=15449], [object_number=16, tag_tid=2, entry_tid=15448]]], state=connected] 2024-08-31T22:16:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=15451' 2024-08-31T22:16:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=15451' 2024-08-31T22:16:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=15451' '!=' 'object_number=18, tag_tid=2, entry_tid=15338' 2024-08-31T22:16:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=15451' = 'object_number=18, tag_tid=2, entry_tid=15338' 2024-08-31T22:16:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=15451' 2024-08-31T22:16:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=15451' 2024-08-31T22:16:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=15451' 2024-08-31T22:16:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15451 2024-08-31T22:16:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 15451 2024-08-31T22:16:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:07.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:07.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:07.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:07.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:07.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:07.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:07.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=15560], [object_number=19, tag_tid=2, entry_tid=15559], [object_number=18, tag_tid=2, entry_tid=15558], [object_number=17, tag_tid=2, entry_tid=15557]]], state=connected] 2024-08-31T22:16:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=15560], [object_number=19, tag_tid=2, entry_tid=15559], [object_number=18, tag_tid=2, entry_tid=15558], [object_number=17, tag_tid=2, entry_tid=15557]]], state=connected] 2024-08-31T22:16:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=15560' 2024-08-31T22:16:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=15560' 2024-08-31T22:16:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=15560' '!=' 'object_number=19, tag_tid=2, entry_tid=15451' 2024-08-31T22:16:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=15560' = 'object_number=19, tag_tid=2, entry_tid=15451' 2024-08-31T22:16:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=15560' 2024-08-31T22:16:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=15560' 2024-08-31T22:16:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=15560' 2024-08-31T22:16:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15560 2024-08-31T22:16:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 15560 2024-08-31T22:16:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:08.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:08.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=15710], [object_number=17, tag_tid=2, entry_tid=15709], [object_number=16, tag_tid=2, entry_tid=15708], [object_number=19, tag_tid=2, entry_tid=15707]]], state=connected] 2024-08-31T22:16:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=15710], [object_number=17, tag_tid=2, entry_tid=15709], [object_number=16, tag_tid=2, entry_tid=15708], [object_number=19, tag_tid=2, entry_tid=15707]]], state=connected] 2024-08-31T22:16:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=15710' 2024-08-31T22:16:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=15710' 2024-08-31T22:16:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=15710' '!=' 'object_number=16, tag_tid=2, entry_tid=15560' 2024-08-31T22:16:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=15710' = 'object_number=16, tag_tid=2, entry_tid=15560' 2024-08-31T22:16:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=15710' 2024-08-31T22:16:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=15710' 2024-08-31T22:16:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:08.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:08.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:08.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:08.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=15710' 2024-08-31T22:16:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:08.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15710 2024-08-31T22:16:08.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 15710 2024-08-31T22:16:08.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:08.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:08.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=15814], [object_number=17, tag_tid=2, entry_tid=15813], [object_number=16, tag_tid=2, entry_tid=15812], [object_number=19, tag_tid=2, entry_tid=15811]]], state=connected] 2024-08-31T22:16:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=15814], [object_number=17, tag_tid=2, entry_tid=15813], [object_number=16, tag_tid=2, entry_tid=15812], [object_number=19, tag_tid=2, entry_tid=15811]]], state=connected] 2024-08-31T22:16:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=15814' 2024-08-31T22:16:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=15814' 2024-08-31T22:16:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=15814' '!=' 'object_number=18, tag_tid=2, entry_tid=15710' 2024-08-31T22:16:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=15814' = 'object_number=18, tag_tid=2, entry_tid=15710' 2024-08-31T22:16:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=15814' 2024-08-31T22:16:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=15814' 2024-08-31T22:16:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=15814' 2024-08-31T22:16:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15814 2024-08-31T22:16:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 15814 2024-08-31T22:16:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=15918], [object_number=17, tag_tid=2, entry_tid=15917], [object_number=16, tag_tid=2, entry_tid=15916], [object_number=19, tag_tid=2, entry_tid=15915]]], state=connected] 2024-08-31T22:16:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=15918], [object_number=17, tag_tid=2, entry_tid=15917], [object_number=16, tag_tid=2, entry_tid=15916], [object_number=19, tag_tid=2, entry_tid=15915]]], state=connected] 2024-08-31T22:16:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=15918' 2024-08-31T22:16:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=15918' 2024-08-31T22:16:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=15918' '!=' 'object_number=18, tag_tid=2, entry_tid=15814' 2024-08-31T22:16:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=15918' = 'object_number=18, tag_tid=2, entry_tid=15814' 2024-08-31T22:16:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=15918' 2024-08-31T22:16:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=15918' 2024-08-31T22:16:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:09.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:09.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:09.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=15918' 2024-08-31T22:16:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15918 2024-08-31T22:16:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 15918 2024-08-31T22:16:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=16023], [object_number=18, tag_tid=2, entry_tid=16022], [object_number=17, tag_tid=2, entry_tid=16021], [object_number=16, tag_tid=2, entry_tid=16020]]], state=connected] 2024-08-31T22:16:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=16023], [object_number=18, tag_tid=2, entry_tid=16022], [object_number=17, tag_tid=2, entry_tid=16021], [object_number=16, tag_tid=2, entry_tid=16020]]], state=connected] 2024-08-31T22:16:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=16023' 2024-08-31T22:16:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=16023' 2024-08-31T22:16:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=16023' '!=' 'object_number=18, tag_tid=2, entry_tid=15918' 2024-08-31T22:16:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=16023' = 'object_number=18, tag_tid=2, entry_tid=15918' 2024-08-31T22:16:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=16023' 2024-08-31T22:16:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=16023' 2024-08-31T22:16:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=16023' 2024-08-31T22:16:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16023 2024-08-31T22:16:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 16023 2024-08-31T22:16:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:09.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:10.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:10.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:10.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=16130], [object_number=17, tag_tid=2, entry_tid=16129], [object_number=16, tag_tid=2, entry_tid=16128], [object_number=19, tag_tid=2, entry_tid=16127]]], state=connected] 2024-08-31T22:16:10.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=16130], [object_number=17, tag_tid=2, entry_tid=16129], [object_number=16, tag_tid=2, entry_tid=16128], [object_number=19, tag_tid=2, entry_tid=16127]]], state=connected] 2024-08-31T22:16:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=16130' 2024-08-31T22:16:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=16130' 2024-08-31T22:16:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=16130' '!=' 'object_number=19, tag_tid=2, entry_tid=16023' 2024-08-31T22:16:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=16130' = 'object_number=19, tag_tid=2, entry_tid=16023' 2024-08-31T22:16:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=16130' 2024-08-31T22:16:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:10.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=16130' 2024-08-31T22:16:10.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:10.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:10.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:10.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:10.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:10.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=16130' 2024-08-31T22:16:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:10.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16130 2024-08-31T22:16:10.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 16130 2024-08-31T22:16:10.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:10.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:10.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:10.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:10.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:10.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:10.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=16242], [object_number=17, tag_tid=2, entry_tid=16241], [object_number=16, tag_tid=2, entry_tid=16240], [object_number=19, tag_tid=2, entry_tid=16239]]], state=connected] 2024-08-31T22:16:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=16242], [object_number=17, tag_tid=2, entry_tid=16241], [object_number=16, tag_tid=2, entry_tid=16240], [object_number=19, tag_tid=2, entry_tid=16239]]], state=connected] 2024-08-31T22:16:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=16242' 2024-08-31T22:16:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=16242' 2024-08-31T22:16:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=16242' '!=' 'object_number=18, tag_tid=2, entry_tid=16130' 2024-08-31T22:16:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=16242' = 'object_number=18, tag_tid=2, entry_tid=16130' 2024-08-31T22:16:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=16242' 2024-08-31T22:16:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=16242' 2024-08-31T22:16:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=16242' 2024-08-31T22:16:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16242 2024-08-31T22:16:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 16242 2024-08-31T22:16:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=16363], [object_number=18, tag_tid=2, entry_tid=16362], [object_number=17, tag_tid=2, entry_tid=16361], [object_number=16, tag_tid=2, entry_tid=16360]]], state=connected] 2024-08-31T22:16:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:11.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:11.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:11.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=16363], [object_number=18, tag_tid=2, entry_tid=16362], [object_number=17, tag_tid=2, entry_tid=16361], [object_number=16, tag_tid=2, entry_tid=16360]]], state=connected] 2024-08-31T22:16:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=16363' 2024-08-31T22:16:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=16363' 2024-08-31T22:16:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=16363' '!=' 'object_number=18, tag_tid=2, entry_tid=16242' 2024-08-31T22:16:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=16363' = 'object_number=18, tag_tid=2, entry_tid=16242' 2024-08-31T22:16:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=16363' 2024-08-31T22:16:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=16363' 2024-08-31T22:16:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=16363' 2024-08-31T22:16:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:11.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16363 2024-08-31T22:16:11.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 16363 2024-08-31T22:16:11.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:11.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:11.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:11.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:11.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:11.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=16468], [object_number=19, tag_tid=2, entry_tid=16467], [object_number=18, tag_tid=2, entry_tid=16466], [object_number=17, tag_tid=2, entry_tid=16465]]], state=connected] 2024-08-31T22:16:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:11.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=16468], [object_number=19, tag_tid=2, entry_tid=16467], [object_number=18, tag_tid=2, entry_tid=16466], [object_number=17, tag_tid=2, entry_tid=16465]]], state=connected] 2024-08-31T22:16:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=16468' 2024-08-31T22:16:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=16468' 2024-08-31T22:16:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=16468' '!=' 'object_number=19, tag_tid=2, entry_tid=16363' 2024-08-31T22:16:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=16468' = 'object_number=19, tag_tid=2, entry_tid=16363' 2024-08-31T22:16:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=16468' 2024-08-31T22:16:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:11.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:11.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:11.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=16468' 2024-08-31T22:16:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=16468' 2024-08-31T22:16:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16468 2024-08-31T22:16:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 16468 2024-08-31T22:16:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:12.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:12.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=16574], [object_number=17, tag_tid=2, entry_tid=16573], [object_number=16, tag_tid=2, entry_tid=16572], [object_number=19, tag_tid=2, entry_tid=16571]]], state=connected] 2024-08-31T22:16:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=16574], [object_number=17, tag_tid=2, entry_tid=16573], [object_number=16, tag_tid=2, entry_tid=16572], [object_number=19, tag_tid=2, entry_tid=16571]]], state=connected] 2024-08-31T22:16:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=16574' 2024-08-31T22:16:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=16574' 2024-08-31T22:16:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=16574' '!=' 'object_number=16, tag_tid=2, entry_tid=16468' 2024-08-31T22:16:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=16574' = 'object_number=16, tag_tid=2, entry_tid=16468' 2024-08-31T22:16:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=16574' 2024-08-31T22:16:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=16574' 2024-08-31T22:16:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=16574' 2024-08-31T22:16:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16574 2024-08-31T22:16:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 16574 2024-08-31T22:16:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=16679], [object_number=18, tag_tid=2, entry_tid=16678], [object_number=17, tag_tid=2, entry_tid=16677], [object_number=16, tag_tid=2, entry_tid=16676]]], state=connected] 2024-08-31T22:16:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:12.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=16679], [object_number=18, tag_tid=2, entry_tid=16678], [object_number=17, tag_tid=2, entry_tid=16677], [object_number=16, tag_tid=2, entry_tid=16676]]], state=connected] 2024-08-31T22:16:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=16679' 2024-08-31T22:16:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=16679' 2024-08-31T22:16:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=16679' '!=' 'object_number=18, tag_tid=2, entry_tid=16574' 2024-08-31T22:16:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=16679' = 'object_number=18, tag_tid=2, entry_tid=16574' 2024-08-31T22:16:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=16679' 2024-08-31T22:16:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=16679' 2024-08-31T22:16:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=16679' 2024-08-31T22:16:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16679 2024-08-31T22:16:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 16679 2024-08-31T22:16:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:12.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:12.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:12.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:12.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:13.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=16785], [object_number=16, tag_tid=2, entry_tid=16784], [object_number=19, tag_tid=2, entry_tid=16783], [object_number=18, tag_tid=2, entry_tid=16782]]], state=connected] 2024-08-31T22:16:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=16785], [object_number=16, tag_tid=2, entry_tid=16784], [object_number=19, tag_tid=2, entry_tid=16783], [object_number=18, tag_tid=2, entry_tid=16782]]], state=connected] 2024-08-31T22:16:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=16785' 2024-08-31T22:16:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=16785' 2024-08-31T22:16:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=16785' '!=' 'object_number=19, tag_tid=2, entry_tid=16679' 2024-08-31T22:16:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=16785' = 'object_number=19, tag_tid=2, entry_tid=16679' 2024-08-31T22:16:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=17, tag_tid=2, entry_tid=16785' 2024-08-31T22:16:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=16785' 2024-08-31T22:16:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=16785' 2024-08-31T22:16:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16785 2024-08-31T22:16:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 16785 2024-08-31T22:16:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:13.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:13.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:13.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:13.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:13.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:13.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:13.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:13.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:13.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:13.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=16887], [object_number=18, tag_tid=2, entry_tid=16886], [object_number=17, tag_tid=2, entry_tid=16885], [object_number=16, tag_tid=2, entry_tid=16884]]], state=connected] 2024-08-31T22:16:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=16887], [object_number=18, tag_tid=2, entry_tid=16886], [object_number=17, tag_tid=2, entry_tid=16885], [object_number=16, tag_tid=2, entry_tid=16884]]], state=connected] 2024-08-31T22:16:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=16887' 2024-08-31T22:16:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=16887' 2024-08-31T22:16:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=16887' '!=' 'object_number=17, tag_tid=2, entry_tid=16785' 2024-08-31T22:16:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=16887' = 'object_number=17, tag_tid=2, entry_tid=16785' 2024-08-31T22:16:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=16887' 2024-08-31T22:16:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:13.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:13.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:13.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=16887' 2024-08-31T22:16:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:13.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=16887' 2024-08-31T22:16:13.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:13.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16887 2024-08-31T22:16:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 16887 2024-08-31T22:16:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:13.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:13.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:13.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=16996], [object_number=19, tag_tid=2, entry_tid=16995], [object_number=18, tag_tid=2, entry_tid=16994], [object_number=17, tag_tid=2, entry_tid=16993]]], state=connected] 2024-08-31T22:16:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=16996], [object_number=19, tag_tid=2, entry_tid=16995], [object_number=18, tag_tid=2, entry_tid=16994], [object_number=17, tag_tid=2, entry_tid=16993]]], state=connected] 2024-08-31T22:16:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=16996' 2024-08-31T22:16:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=16996' 2024-08-31T22:16:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=16996' '!=' 'object_number=19, tag_tid=2, entry_tid=16887' 2024-08-31T22:16:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=16996' = 'object_number=19, tag_tid=2, entry_tid=16887' 2024-08-31T22:16:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=16996' 2024-08-31T22:16:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=16996' 2024-08-31T22:16:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=16996' 2024-08-31T22:16:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16996 2024-08-31T22:16:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 16996 2024-08-31T22:16:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:14.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:14.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:14.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=17044], [object_number=23, tag_tid=2, entry_tid=17043], [object_number=22, tag_tid=2, entry_tid=17042], [object_number=17, tag_tid=2, entry_tid=17041]]], state=connected] 2024-08-31T22:16:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:14.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 4 2024-08-31T22:16:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=17044], [object_number=23, tag_tid=2, entry_tid=17043], [object_number=22, tag_tid=2, entry_tid=17042], [object_number=17, tag_tid=2, entry_tid=17041]]], state=connected] 2024-08-31T22:16:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=17044' 2024-08-31T22:16:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=17044' 2024-08-31T22:16:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=17044' '!=' 'object_number=16, tag_tid=2, entry_tid=16996' 2024-08-31T22:16:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=17044' = 'object_number=16, tag_tid=2, entry_tid=16996' 2024-08-31T22:16:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=17044' 2024-08-31T22:16:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=17044' 2024-08-31T22:16:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:14.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=17044' 2024-08-31T22:16:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17044 2024-08-31T22:16:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 17044 2024-08-31T22:16:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:14.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:14.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=17137], [object_number=20, tag_tid=2, entry_tid=17136], [object_number=23, tag_tid=2, entry_tid=17135], [object_number=22, tag_tid=2, entry_tid=17134]]], state=connected] 2024-08-31T22:16:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:15.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:15.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:15.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:15.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=17137], [object_number=20, tag_tid=2, entry_tid=17136], [object_number=23, tag_tid=2, entry_tid=17135], [object_number=22, tag_tid=2, entry_tid=17134]]], state=connected] 2024-08-31T22:16:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=17137' 2024-08-31T22:16:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=17137' 2024-08-31T22:16:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=17137' '!=' 'object_number=20, tag_tid=2, entry_tid=17044' 2024-08-31T22:16:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=17137' = 'object_number=20, tag_tid=2, entry_tid=17044' 2024-08-31T22:16:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=17137' 2024-08-31T22:16:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=17137' 2024-08-31T22:16:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=17137' 2024-08-31T22:16:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17137 2024-08-31T22:16:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 17137 2024-08-31T22:16:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:15.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:15.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:15.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:15.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:15.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:15.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:15.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:15.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:15.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:15.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:15.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:15.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:15.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:15.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=17250], [object_number=21, tag_tid=2, entry_tid=17249], [object_number=20, tag_tid=2, entry_tid=17248], [object_number=23, tag_tid=2, entry_tid=17247]]], state=connected] 2024-08-31T22:16:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:15.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=17250], [object_number=21, tag_tid=2, entry_tid=17249], [object_number=20, tag_tid=2, entry_tid=17248], [object_number=23, tag_tid=2, entry_tid=17247]]], state=connected] 2024-08-31T22:16:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=17250' 2024-08-31T22:16:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=17250' 2024-08-31T22:16:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=17250' '!=' 'object_number=21, tag_tid=2, entry_tid=17137' 2024-08-31T22:16:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=17250' = 'object_number=21, tag_tid=2, entry_tid=17137' 2024-08-31T22:16:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=17250' 2024-08-31T22:16:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=17250' 2024-08-31T22:16:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=17250' 2024-08-31T22:16:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17250 2024-08-31T22:16:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 17250 2024-08-31T22:16:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=17408], [object_number=23, tag_tid=2, entry_tid=17407], [object_number=22, tag_tid=2, entry_tid=17406], [object_number=21, tag_tid=2, entry_tid=17405]]], state=connected] 2024-08-31T22:16:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=17408], [object_number=23, tag_tid=2, entry_tid=17407], [object_number=22, tag_tid=2, entry_tid=17406], [object_number=21, tag_tid=2, entry_tid=17405]]], state=connected] 2024-08-31T22:16:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:16.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=17408' 2024-08-31T22:16:16.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=17408' 2024-08-31T22:16:16.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=17408' '!=' 'object_number=22, tag_tid=2, entry_tid=17250' 2024-08-31T22:16:16.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:16.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=17408' = 'object_number=22, tag_tid=2, entry_tid=17250' 2024-08-31T22:16:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=17408' 2024-08-31T22:16:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=17408' 2024-08-31T22:16:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:16.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:16.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=17408' 2024-08-31T22:16:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17408 2024-08-31T22:16:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 17408 2024-08-31T22:16:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:16.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:16.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:16.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=17512], [object_number=23, tag_tid=2, entry_tid=17511], [object_number=22, tag_tid=2, entry_tid=17510], [object_number=21, tag_tid=2, entry_tid=17509]]], state=connected] 2024-08-31T22:16:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=17512], [object_number=23, tag_tid=2, entry_tid=17511], [object_number=22, tag_tid=2, entry_tid=17510], [object_number=21, tag_tid=2, entry_tid=17509]]], state=connected] 2024-08-31T22:16:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=17512' 2024-08-31T22:16:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=17512' 2024-08-31T22:16:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=17512' '!=' 'object_number=20, tag_tid=2, entry_tid=17408' 2024-08-31T22:16:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=17512' = 'object_number=20, tag_tid=2, entry_tid=17408' 2024-08-31T22:16:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=17512' 2024-08-31T22:16:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=17512' 2024-08-31T22:16:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=17512' 2024-08-31T22:16:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17512 2024-08-31T22:16:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 17512 2024-08-31T22:16:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:17.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:17.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=17622], [object_number=21, tag_tid=2, entry_tid=17621], [object_number=20, tag_tid=2, entry_tid=17620], [object_number=23, tag_tid=2, entry_tid=17619]]], state=connected] 2024-08-31T22:16:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=17622], [object_number=21, tag_tid=2, entry_tid=17621], [object_number=20, tag_tid=2, entry_tid=17620], [object_number=23, tag_tid=2, entry_tid=17619]]], state=connected] 2024-08-31T22:16:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=17622' 2024-08-31T22:16:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=17622' 2024-08-31T22:16:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=17622' '!=' 'object_number=20, tag_tid=2, entry_tid=17512' 2024-08-31T22:16:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=17622' = 'object_number=20, tag_tid=2, entry_tid=17512' 2024-08-31T22:16:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=17622' 2024-08-31T22:16:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=17622' 2024-08-31T22:16:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=17622' 2024-08-31T22:16:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17622 2024-08-31T22:16:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 17622 2024-08-31T22:16:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=17734], [object_number=21, tag_tid=2, entry_tid=17733], [object_number=20, tag_tid=2, entry_tid=17732], [object_number=23, tag_tid=2, entry_tid=17731]]], state=connected] 2024-08-31T22:16:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=17734], [object_number=21, tag_tid=2, entry_tid=17733], [object_number=20, tag_tid=2, entry_tid=17732], [object_number=23, tag_tid=2, entry_tid=17731]]], state=connected] 2024-08-31T22:16:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=17734' 2024-08-31T22:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=17734' 2024-08-31T22:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=17734' '!=' 'object_number=22, tag_tid=2, entry_tid=17622' 2024-08-31T22:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=17734' = 'object_number=22, tag_tid=2, entry_tid=17622' 2024-08-31T22:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=17734' 2024-08-31T22:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=17734' 2024-08-31T22:16:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=17734' 2024-08-31T22:16:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:17.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17734 2024-08-31T22:16:17.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 17734 2024-08-31T22:16:17.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:17.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:17.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=17851], [object_number=22, tag_tid=2, entry_tid=17850], [object_number=21, tag_tid=2, entry_tid=17849], [object_number=20, tag_tid=2, entry_tid=17848]]], state=connected] 2024-08-31T22:16:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:18.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=17851], [object_number=22, tag_tid=2, entry_tid=17850], [object_number=21, tag_tid=2, entry_tid=17849], [object_number=20, tag_tid=2, entry_tid=17848]]], state=connected] 2024-08-31T22:16:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=17851' 2024-08-31T22:16:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=17851' 2024-08-31T22:16:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=17851' '!=' 'object_number=22, tag_tid=2, entry_tid=17734' 2024-08-31T22:16:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=17851' = 'object_number=22, tag_tid=2, entry_tid=17734' 2024-08-31T22:16:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=17851' 2024-08-31T22:16:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:18.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=17851' 2024-08-31T22:16:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=17851' 2024-08-31T22:16:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17851 2024-08-31T22:16:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 17851 2024-08-31T22:16:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:18.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:18.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:18.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=17969], [object_number=20, tag_tid=2, entry_tid=17968], [object_number=23, tag_tid=2, entry_tid=17967], [object_number=22, tag_tid=2, entry_tid=17966]]], state=connected] 2024-08-31T22:16:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=17969], [object_number=20, tag_tid=2, entry_tid=17968], [object_number=23, tag_tid=2, entry_tid=17967], [object_number=22, tag_tid=2, entry_tid=17966]]], state=connected] 2024-08-31T22:16:18.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=17969' 2024-08-31T22:16:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=17969' 2024-08-31T22:16:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=17969' '!=' 'object_number=23, tag_tid=2, entry_tid=17851' 2024-08-31T22:16:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=17969' = 'object_number=23, tag_tid=2, entry_tid=17851' 2024-08-31T22:16:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=17969' 2024-08-31T22:16:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=17969' 2024-08-31T22:16:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=17969' 2024-08-31T22:16:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17969 2024-08-31T22:16:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 17969 2024-08-31T22:16:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:19.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:19.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=18089], [object_number=20, tag_tid=2, entry_tid=18088], [object_number=23, tag_tid=2, entry_tid=18087], [object_number=22, tag_tid=2, entry_tid=18086]]], state=connected] 2024-08-31T22:16:19.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=18089], [object_number=20, tag_tid=2, entry_tid=18088], [object_number=23, tag_tid=2, entry_tid=18087], [object_number=22, tag_tid=2, entry_tid=18086]]], state=connected] 2024-08-31T22:16:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=18089' 2024-08-31T22:16:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=18089' 2024-08-31T22:16:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=18089' '!=' 'object_number=21, tag_tid=2, entry_tid=17969' 2024-08-31T22:16:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=18089' = 'object_number=21, tag_tid=2, entry_tid=17969' 2024-08-31T22:16:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=18089' 2024-08-31T22:16:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=18089' 2024-08-31T22:16:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=18089' 2024-08-31T22:16:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18089 2024-08-31T22:16:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 18089 2024-08-31T22:16:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:19.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:19.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:19.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:19.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:19.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:19.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:19.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:19.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:19.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:19.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=18209], [object_number=20, tag_tid=2, entry_tid=18208], [object_number=23, tag_tid=2, entry_tid=18207], [object_number=22, tag_tid=2, entry_tid=18206]]], state=connected] 2024-08-31T22:16:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=18209], [object_number=20, tag_tid=2, entry_tid=18208], [object_number=23, tag_tid=2, entry_tid=18207], [object_number=22, tag_tid=2, entry_tid=18206]]], state=connected] 2024-08-31T22:16:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=18209' 2024-08-31T22:16:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=18209' 2024-08-31T22:16:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=18209' '!=' 'object_number=21, tag_tid=2, entry_tid=18089' 2024-08-31T22:16:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=18209' = 'object_number=21, tag_tid=2, entry_tid=18089' 2024-08-31T22:16:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=18209' 2024-08-31T22:16:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=18209' 2024-08-31T22:16:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:19.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:19.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=18209' 2024-08-31T22:16:19.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:19.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18209 2024-08-31T22:16:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 18209 2024-08-31T22:16:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=18319], [object_number=22, tag_tid=2, entry_tid=18318], [object_number=21, tag_tid=2, entry_tid=18317], [object_number=20, tag_tid=2, entry_tid=18316]]], state=connected] 2024-08-31T22:16:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=18319], [object_number=22, tag_tid=2, entry_tid=18318], [object_number=21, tag_tid=2, entry_tid=18317], [object_number=20, tag_tid=2, entry_tid=18316]]], state=connected] 2024-08-31T22:16:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=18319' 2024-08-31T22:16:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=18319' 2024-08-31T22:16:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=18319' '!=' 'object_number=21, tag_tid=2, entry_tid=18209' 2024-08-31T22:16:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=18319' = 'object_number=21, tag_tid=2, entry_tid=18209' 2024-08-31T22:16:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=18319' 2024-08-31T22:16:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=18319' 2024-08-31T22:16:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=18319' 2024-08-31T22:16:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18319 2024-08-31T22:16:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 18319 2024-08-31T22:16:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:20.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:20.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:20.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=18430], [object_number=21, tag_tid=2, entry_tid=18429], [object_number=20, tag_tid=2, entry_tid=18428], [object_number=23, tag_tid=2, entry_tid=18427]]], state=connected] 2024-08-31T22:16:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:20.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=18430], [object_number=21, tag_tid=2, entry_tid=18429], [object_number=20, tag_tid=2, entry_tid=18428], [object_number=23, tag_tid=2, entry_tid=18427]]], state=connected] 2024-08-31T22:16:20.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=18430' 2024-08-31T22:16:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=18430' 2024-08-31T22:16:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=18430' '!=' 'object_number=23, tag_tid=2, entry_tid=18319' 2024-08-31T22:16:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=18430' = 'object_number=23, tag_tid=2, entry_tid=18319' 2024-08-31T22:16:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=18430' 2024-08-31T22:16:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=18430' 2024-08-31T22:16:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:20.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=18430' 2024-08-31T22:16:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18430 2024-08-31T22:16:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 18430 2024-08-31T22:16:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:20.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:20.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:20.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=18507], [object_number=22, tag_tid=2, entry_tid=18506], [object_number=21, tag_tid=2, entry_tid=18505], [object_number=20, tag_tid=2, entry_tid=18504]]], state=connected] 2024-08-31T22:16:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:21.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:21.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:21.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:21.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=18507], [object_number=22, tag_tid=2, entry_tid=18506], [object_number=21, tag_tid=2, entry_tid=18505], [object_number=20, tag_tid=2, entry_tid=18504]]], state=connected] 2024-08-31T22:16:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=18507' 2024-08-31T22:16:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=18507' 2024-08-31T22:16:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=18507' '!=' 'object_number=22, tag_tid=2, entry_tid=18430' 2024-08-31T22:16:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=18507' = 'object_number=22, tag_tid=2, entry_tid=18430' 2024-08-31T22:16:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=18507' 2024-08-31T22:16:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=18507' 2024-08-31T22:16:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=18507' 2024-08-31T22:16:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18507 2024-08-31T22:16:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 18507 2024-08-31T22:16:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=18620], [object_number=23, tag_tid=2, entry_tid=18619], [object_number=22, tag_tid=2, entry_tid=18618], [object_number=21, tag_tid=2, entry_tid=18617]]], state=connected] 2024-08-31T22:16:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:21.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:21.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:21.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=18620], [object_number=23, tag_tid=2, entry_tid=18619], [object_number=22, tag_tid=2, entry_tid=18618], [object_number=21, tag_tid=2, entry_tid=18617]]], state=connected] 2024-08-31T22:16:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=18620' 2024-08-31T22:16:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=18620' 2024-08-31T22:16:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=18620' '!=' 'object_number=23, tag_tid=2, entry_tid=18507' 2024-08-31T22:16:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=18620' = 'object_number=23, tag_tid=2, entry_tid=18507' 2024-08-31T22:16:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=18620' 2024-08-31T22:16:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=18620' 2024-08-31T22:16:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=18620' 2024-08-31T22:16:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18620 2024-08-31T22:16:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 18620 2024-08-31T22:16:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=18743], [object_number=22, tag_tid=2, entry_tid=18742], [object_number=21, tag_tid=2, entry_tid=18741], [object_number=20, tag_tid=2, entry_tid=18740]]], state=connected] 2024-08-31T22:16:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=18743], [object_number=22, tag_tid=2, entry_tid=18742], [object_number=21, tag_tid=2, entry_tid=18741], [object_number=20, tag_tid=2, entry_tid=18740]]], state=connected] 2024-08-31T22:16:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=18743' 2024-08-31T22:16:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=18743' 2024-08-31T22:16:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=18743' '!=' 'object_number=20, tag_tid=2, entry_tid=18620' 2024-08-31T22:16:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=18743' = 'object_number=20, tag_tid=2, entry_tid=18620' 2024-08-31T22:16:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=18743' 2024-08-31T22:16:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=18743' 2024-08-31T22:16:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=18743' 2024-08-31T22:16:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18743 2024-08-31T22:16:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 18743 2024-08-31T22:16:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=18863], [object_number=22, tag_tid=2, entry_tid=18862], [object_number=21, tag_tid=2, entry_tid=18861], [object_number=20, tag_tid=2, entry_tid=18860]]], state=connected] 2024-08-31T22:16:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=18863], [object_number=22, tag_tid=2, entry_tid=18862], [object_number=21, tag_tid=2, entry_tid=18861], [object_number=20, tag_tid=2, entry_tid=18860]]], state=connected] 2024-08-31T22:16:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=18863' 2024-08-31T22:16:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=18863' 2024-08-31T22:16:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=18863' '!=' 'object_number=23, tag_tid=2, entry_tid=18743' 2024-08-31T22:16:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=18863' = 'object_number=23, tag_tid=2, entry_tid=18743' 2024-08-31T22:16:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=18863' 2024-08-31T22:16:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=18863' 2024-08-31T22:16:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=18863' 2024-08-31T22:16:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18863 2024-08-31T22:16:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 18863 2024-08-31T22:16:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=18980], [object_number=23, tag_tid=2, entry_tid=18979], [object_number=22, tag_tid=2, entry_tid=18978], [object_number=21, tag_tid=2, entry_tid=18977]]], state=connected] 2024-08-31T22:16:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=18980], [object_number=23, tag_tid=2, entry_tid=18979], [object_number=22, tag_tid=2, entry_tid=18978], [object_number=21, tag_tid=2, entry_tid=18977]]], state=connected] 2024-08-31T22:16:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=18980' 2024-08-31T22:16:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=18980' 2024-08-31T22:16:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=18980' '!=' 'object_number=23, tag_tid=2, entry_tid=18863' 2024-08-31T22:16:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=18980' = 'object_number=23, tag_tid=2, entry_tid=18863' 2024-08-31T22:16:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=18980' 2024-08-31T22:16:23.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:23.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:23.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:23.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=18980' 2024-08-31T22:16:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=18980' 2024-08-31T22:16:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:23.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18980 2024-08-31T22:16:23.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 18980 2024-08-31T22:16:23.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:23.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:23.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:23.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=19100], [object_number=23, tag_tid=2, entry_tid=19099], [object_number=22, tag_tid=2, entry_tid=19098], [object_number=21, tag_tid=2, entry_tid=19097]]], state=connected] 2024-08-31T22:16:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=19100], [object_number=23, tag_tid=2, entry_tid=19099], [object_number=22, tag_tid=2, entry_tid=19098], [object_number=21, tag_tid=2, entry_tid=19097]]], state=connected] 2024-08-31T22:16:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=19100' 2024-08-31T22:16:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=19100' 2024-08-31T22:16:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=19100' '!=' 'object_number=20, tag_tid=2, entry_tid=18980' 2024-08-31T22:16:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=19100' = 'object_number=20, tag_tid=2, entry_tid=18980' 2024-08-31T22:16:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=19100' 2024-08-31T22:16:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=19100' 2024-08-31T22:16:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:23.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:23.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:23.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=19100' 2024-08-31T22:16:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19100 2024-08-31T22:16:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 19100 2024-08-31T22:16:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=19204], [object_number=23, tag_tid=2, entry_tid=19203], [object_number=22, tag_tid=2, entry_tid=19202], [object_number=21, tag_tid=2, entry_tid=19201]]], state=connected] 2024-08-31T22:16:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=19204], [object_number=23, tag_tid=2, entry_tid=19203], [object_number=22, tag_tid=2, entry_tid=19202], [object_number=21, tag_tid=2, entry_tid=19201]]], state=connected] 2024-08-31T22:16:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=19204' 2024-08-31T22:16:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=19204' 2024-08-31T22:16:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=19204' '!=' 'object_number=20, tag_tid=2, entry_tid=19100' 2024-08-31T22:16:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=19204' = 'object_number=20, tag_tid=2, entry_tid=19100' 2024-08-31T22:16:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=19204' 2024-08-31T22:16:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=19204' 2024-08-31T22:16:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=19204' 2024-08-31T22:16:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19204 2024-08-31T22:16:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 19204 2024-08-31T22:16:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:24.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:24.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:24.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:24.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=19312], [object_number=23, tag_tid=2, entry_tid=19311], [object_number=22, tag_tid=2, entry_tid=19310], [object_number=21, tag_tid=2, entry_tid=19309]]], state=connected] 2024-08-31T22:16:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=19312], [object_number=23, tag_tid=2, entry_tid=19311], [object_number=22, tag_tid=2, entry_tid=19310], [object_number=21, tag_tid=2, entry_tid=19309]]], state=connected] 2024-08-31T22:16:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=19312' 2024-08-31T22:16:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=19312' 2024-08-31T22:16:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=19312' '!=' 'object_number=20, tag_tid=2, entry_tid=19204' 2024-08-31T22:16:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=19312' = 'object_number=20, tag_tid=2, entry_tid=19204' 2024-08-31T22:16:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=19312' 2024-08-31T22:16:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=19312' 2024-08-31T22:16:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=19312' 2024-08-31T22:16:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19312 2024-08-31T22:16:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 19312 2024-08-31T22:16:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=19425], [object_number=20, tag_tid=2, entry_tid=19424], [object_number=23, tag_tid=2, entry_tid=19423], [object_number=22, tag_tid=2, entry_tid=19422]]], state=connected] 2024-08-31T22:16:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=19425], [object_number=20, tag_tid=2, entry_tid=19424], [object_number=23, tag_tid=2, entry_tid=19423], [object_number=22, tag_tid=2, entry_tid=19422]]], state=connected] 2024-08-31T22:16:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=19425' 2024-08-31T22:16:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=19425' 2024-08-31T22:16:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=19425' '!=' 'object_number=20, tag_tid=2, entry_tid=19312' 2024-08-31T22:16:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=19425' = 'object_number=20, tag_tid=2, entry_tid=19312' 2024-08-31T22:16:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=19425' 2024-08-31T22:16:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=19425' 2024-08-31T22:16:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=19425' 2024-08-31T22:16:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19425 2024-08-31T22:16:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 19425 2024-08-31T22:16:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:25.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:25.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=19543], [object_number=22, tag_tid=2, entry_tid=19542], [object_number=21, tag_tid=2, entry_tid=19541], [object_number=20, tag_tid=2, entry_tid=19540]]], state=connected] 2024-08-31T22:16:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=19543], [object_number=22, tag_tid=2, entry_tid=19542], [object_number=21, tag_tid=2, entry_tid=19541], [object_number=20, tag_tid=2, entry_tid=19540]]], state=connected] 2024-08-31T22:16:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=19543' 2024-08-31T22:16:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=19543' 2024-08-31T22:16:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=19543' '!=' 'object_number=21, tag_tid=2, entry_tid=19425' 2024-08-31T22:16:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=19543' = 'object_number=21, tag_tid=2, entry_tid=19425' 2024-08-31T22:16:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=19543' 2024-08-31T22:16:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=19543' 2024-08-31T22:16:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=19543' 2024-08-31T22:16:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19543 2024-08-31T22:16:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 19543 2024-08-31T22:16:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=19670], [object_number=21, tag_tid=2, entry_tid=19669], [object_number=20, tag_tid=2, entry_tid=19668], [object_number=23, tag_tid=2, entry_tid=19667]]], state=connected] 2024-08-31T22:16:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:26.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:26.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:26.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:26.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:26.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=19670], [object_number=21, tag_tid=2, entry_tid=19669], [object_number=20, tag_tid=2, entry_tid=19668], [object_number=23, tag_tid=2, entry_tid=19667]]], state=connected] 2024-08-31T22:16:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=19670' 2024-08-31T22:16:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=19670' 2024-08-31T22:16:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=19670' '!=' 'object_number=23, tag_tid=2, entry_tid=19543' 2024-08-31T22:16:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=19670' = 'object_number=23, tag_tid=2, entry_tid=19543' 2024-08-31T22:16:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=19670' 2024-08-31T22:16:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=19670' 2024-08-31T22:16:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=19670' 2024-08-31T22:16:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:26.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19670 2024-08-31T22:16:26.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 19670 2024-08-31T22:16:26.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:26.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:26.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:26.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:26.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=19792], [object_number=23, tag_tid=2, entry_tid=19791], [object_number=22, tag_tid=2, entry_tid=19790], [object_number=21, tag_tid=2, entry_tid=19789]]], state=connected] 2024-08-31T22:16:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=19792], [object_number=23, tag_tid=2, entry_tid=19791], [object_number=22, tag_tid=2, entry_tid=19790], [object_number=21, tag_tid=2, entry_tid=19789]]], state=connected] 2024-08-31T22:16:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=19792' 2024-08-31T22:16:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=19792' 2024-08-31T22:16:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=19792' '!=' 'object_number=22, tag_tid=2, entry_tid=19670' 2024-08-31T22:16:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=19792' = 'object_number=22, tag_tid=2, entry_tid=19670' 2024-08-31T22:16:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=19792' 2024-08-31T22:16:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=19792' 2024-08-31T22:16:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=19792' 2024-08-31T22:16:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19792 2024-08-31T22:16:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 19792 2024-08-31T22:16:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:26.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:26.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:26.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=19915], [object_number=22, tag_tid=2, entry_tid=19914], [object_number=21, tag_tid=2, entry_tid=19913], [object_number=20, tag_tid=2, entry_tid=19912]]], state=connected] 2024-08-31T22:16:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=19915], [object_number=22, tag_tid=2, entry_tid=19914], [object_number=21, tag_tid=2, entry_tid=19913], [object_number=20, tag_tid=2, entry_tid=19912]]], state=connected] 2024-08-31T22:16:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=19915' 2024-08-31T22:16:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=19915' 2024-08-31T22:16:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=19915' '!=' 'object_number=20, tag_tid=2, entry_tid=19792' 2024-08-31T22:16:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=19915' = 'object_number=20, tag_tid=2, entry_tid=19792' 2024-08-31T22:16:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=19915' 2024-08-31T22:16:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=19915' 2024-08-31T22:16:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=19915' 2024-08-31T22:16:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19915 2024-08-31T22:16:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 19915 2024-08-31T22:16:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:27.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:27.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:27.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=20035], [object_number=22, tag_tid=2, entry_tid=20034], [object_number=21, tag_tid=2, entry_tid=20033], [object_number=20, tag_tid=2, entry_tid=20032]]], state=connected] 2024-08-31T22:16:27.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=20035], [object_number=22, tag_tid=2, entry_tid=20034], [object_number=21, tag_tid=2, entry_tid=20033], [object_number=20, tag_tid=2, entry_tid=20032]]], state=connected] 2024-08-31T22:16:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=20035' 2024-08-31T22:16:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=20035' 2024-08-31T22:16:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=20035' '!=' 'object_number=23, tag_tid=2, entry_tid=19915' 2024-08-31T22:16:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=20035' = 'object_number=23, tag_tid=2, entry_tid=19915' 2024-08-31T22:16:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=20035' 2024-08-31T22:16:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=20035' 2024-08-31T22:16:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=20035' 2024-08-31T22:16:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20035 2024-08-31T22:16:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 20035 2024-08-31T22:16:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:27.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:27.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:27.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=20155], [object_number=22, tag_tid=2, entry_tid=20154], [object_number=21, tag_tid=2, entry_tid=20153], [object_number=20, tag_tid=2, entry_tid=20152]]], state=connected] 2024-08-31T22:16:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=20155], [object_number=22, tag_tid=2, entry_tid=20154], [object_number=21, tag_tid=2, entry_tid=20153], [object_number=20, tag_tid=2, entry_tid=20152]]], state=connected] 2024-08-31T22:16:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=20155' 2024-08-31T22:16:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=20155' 2024-08-31T22:16:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=20155' '!=' 'object_number=23, tag_tid=2, entry_tid=20035' 2024-08-31T22:16:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=20155' = 'object_number=23, tag_tid=2, entry_tid=20035' 2024-08-31T22:16:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=20155' 2024-08-31T22:16:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=20155' 2024-08-31T22:16:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=20155' 2024-08-31T22:16:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20155 2024-08-31T22:16:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 20155 2024-08-31T22:16:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:28.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:28.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:28.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:28.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:28.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:28.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:28.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:28.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=20274], [object_number=21, tag_tid=2, entry_tid=20273], [object_number=20, tag_tid=2, entry_tid=20272], [object_number=23, tag_tid=2, entry_tid=20271]]], state=connected] 2024-08-31T22:16:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:28.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:28.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:28.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:28.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:28.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:28.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:28.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:28.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=20274], [object_number=21, tag_tid=2, entry_tid=20273], [object_number=20, tag_tid=2, entry_tid=20272], [object_number=23, tag_tid=2, entry_tid=20271]]], state=connected] 2024-08-31T22:16:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=20274' 2024-08-31T22:16:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=20274' 2024-08-31T22:16:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=20274' '!=' 'object_number=23, tag_tid=2, entry_tid=20155' 2024-08-31T22:16:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=20274' = 'object_number=23, tag_tid=2, entry_tid=20155' 2024-08-31T22:16:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=20274' 2024-08-31T22:16:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=20274' 2024-08-31T22:16:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=20274' 2024-08-31T22:16:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20274 2024-08-31T22:16:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 20274 2024-08-31T22:16:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=20433], [object_number=20, tag_tid=2, entry_tid=20432], [object_number=23, tag_tid=2, entry_tid=20431], [object_number=22, tag_tid=2, entry_tid=20430]]], state=connected] 2024-08-31T22:16:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:29.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:29.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:29.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:29.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:29.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=20433], [object_number=20, tag_tid=2, entry_tid=20432], [object_number=23, tag_tid=2, entry_tid=20431], [object_number=22, tag_tid=2, entry_tid=20430]]], state=connected] 2024-08-31T22:16:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=20433' 2024-08-31T22:16:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=20433' 2024-08-31T22:16:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=20433' '!=' 'object_number=22, tag_tid=2, entry_tid=20274' 2024-08-31T22:16:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=20433' = 'object_number=22, tag_tid=2, entry_tid=20274' 2024-08-31T22:16:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=20433' 2024-08-31T22:16:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=20433' 2024-08-31T22:16:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=20433' 2024-08-31T22:16:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20433 2024-08-31T22:16:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 20433 2024-08-31T22:16:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:29.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:29.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:29.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:29.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:29.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:29.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:29.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:29.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:29.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:29.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:29.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:29.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=20519], [object_number=22, tag_tid=2, entry_tid=20518], [object_number=21, tag_tid=2, entry_tid=20517], [object_number=20, tag_tid=2, entry_tid=20516]]], state=connected] 2024-08-31T22:16:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 5 2024-08-31T22:16:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=20519], [object_number=22, tag_tid=2, entry_tid=20518], [object_number=21, tag_tid=2, entry_tid=20517], [object_number=20, tag_tid=2, entry_tid=20516]]], state=connected] 2024-08-31T22:16:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=20519' 2024-08-31T22:16:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=20519' 2024-08-31T22:16:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=20519' '!=' 'object_number=21, tag_tid=2, entry_tid=20433' 2024-08-31T22:16:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=20519' = 'object_number=21, tag_tid=2, entry_tid=20433' 2024-08-31T22:16:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=20519' 2024-08-31T22:16:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=20519' 2024-08-31T22:16:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=20519' 2024-08-31T22:16:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20519 2024-08-31T22:16:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 20519 2024-08-31T22:16:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=20577], [object_number=24, tag_tid=2, entry_tid=20576], [object_number=27, tag_tid=2, entry_tid=20575], [object_number=26, tag_tid=2, entry_tid=20574]]], state=connected] 2024-08-31T22:16:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=20577], [object_number=24, tag_tid=2, entry_tid=20576], [object_number=27, tag_tid=2, entry_tid=20575], [object_number=26, tag_tid=2, entry_tid=20574]]], state=connected] 2024-08-31T22:16:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=20577' 2024-08-31T22:16:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=20577' 2024-08-31T22:16:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=20577' '!=' 'object_number=23, tag_tid=2, entry_tid=20519' 2024-08-31T22:16:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=20577' = 'object_number=23, tag_tid=2, entry_tid=20519' 2024-08-31T22:16:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=25, tag_tid=2, entry_tid=20577' 2024-08-31T22:16:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=20577' 2024-08-31T22:16:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=20577' 2024-08-31T22:16:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20577 2024-08-31T22:16:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 20577 2024-08-31T22:16:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:30.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:30.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:30.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:30.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:30.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=27, tag_tid=2, entry_tid=20739], [object_number=26, tag_tid=2, entry_tid=20738], [object_number=25, tag_tid=2, entry_tid=20737], [object_number=24, tag_tid=2, entry_tid=20736]]], state=connected] 2024-08-31T22:16:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=27, tag_tid=2, entry_tid=20739], [object_number=26, tag_tid=2, entry_tid=20738], [object_number=25, tag_tid=2, entry_tid=20737], [object_number=24, tag_tid=2, entry_tid=20736]]], state=connected] 2024-08-31T22:16:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=27, tag_tid=2, entry_tid=20739' 2024-08-31T22:16:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=27, tag_tid=2, entry_tid=20739' 2024-08-31T22:16:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=27, tag_tid=2, entry_tid=20739' '!=' 'object_number=25, tag_tid=2, entry_tid=20577' 2024-08-31T22:16:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=27, tag_tid=2, entry_tid=20739' = 'object_number=25, tag_tid=2, entry_tid=20577' 2024-08-31T22:16:30.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=27, tag_tid=2, entry_tid=20739' 2024-08-31T22:16:30.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:30.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:30.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:30.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:30.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=27, tag_tid=2, entry_tid=20739' 2024-08-31T22:16:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:30.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=27, tag_tid=2, entry_tid=20739' 2024-08-31T22:16:30.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:30.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20739 2024-08-31T22:16:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 20739 2024-08-31T22:16:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:31.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:31.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=20841], [object_number=24, tag_tid=2, entry_tid=20840], [object_number=27, tag_tid=2, entry_tid=20839], [object_number=26, tag_tid=2, entry_tid=20838]]], state=connected] 2024-08-31T22:16:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=20841], [object_number=24, tag_tid=2, entry_tid=20840], [object_number=27, tag_tid=2, entry_tid=20839], [object_number=26, tag_tid=2, entry_tid=20838]]], state=connected] 2024-08-31T22:16:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=20841' 2024-08-31T22:16:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=20841' 2024-08-31T22:16:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=20841' '!=' 'object_number=27, tag_tid=2, entry_tid=20739' 2024-08-31T22:16:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=20841' = 'object_number=27, tag_tid=2, entry_tid=20739' 2024-08-31T22:16:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=25, tag_tid=2, entry_tid=20841' 2024-08-31T22:16:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=20841' 2024-08-31T22:16:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=20841' 2024-08-31T22:16:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20841 2024-08-31T22:16:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 20841 2024-08-31T22:16:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:31.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:31.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:31.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:31.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:31.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=20940], [object_number=27, tag_tid=2, entry_tid=20939], [object_number=26, tag_tid=2, entry_tid=20938], [object_number=25, tag_tid=2, entry_tid=20937]]], state=connected] 2024-08-31T22:16:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=20940], [object_number=27, tag_tid=2, entry_tid=20939], [object_number=26, tag_tid=2, entry_tid=20938], [object_number=25, tag_tid=2, entry_tid=20937]]], state=connected] 2024-08-31T22:16:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=24, tag_tid=2, entry_tid=20940' 2024-08-31T22:16:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=24, tag_tid=2, entry_tid=20940' 2024-08-31T22:16:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=20940' '!=' 'object_number=25, tag_tid=2, entry_tid=20841' 2024-08-31T22:16:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=20940' = 'object_number=25, tag_tid=2, entry_tid=20841' 2024-08-31T22:16:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=24, tag_tid=2, entry_tid=20940' 2024-08-31T22:16:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=20940' 2024-08-31T22:16:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=20940' 2024-08-31T22:16:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20940 2024-08-31T22:16:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 20940 2024-08-31T22:16:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=21041], [object_number=24, tag_tid=2, entry_tid=21040], [object_number=27, tag_tid=2, entry_tid=21039], [object_number=26, tag_tid=2, entry_tid=21038]]], state=connected] 2024-08-31T22:16:32.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=21041], [object_number=24, tag_tid=2, entry_tid=21040], [object_number=27, tag_tid=2, entry_tid=21039], [object_number=26, tag_tid=2, entry_tid=21038]]], state=connected] 2024-08-31T22:16:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:32.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=21041' 2024-08-31T22:16:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=21041' 2024-08-31T22:16:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=21041' '!=' 'object_number=24, tag_tid=2, entry_tid=20940' 2024-08-31T22:16:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=21041' = 'object_number=24, tag_tid=2, entry_tid=20940' 2024-08-31T22:16:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=25, tag_tid=2, entry_tid=21041' 2024-08-31T22:16:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=21041' 2024-08-31T22:16:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=21041' 2024-08-31T22:16:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21041 2024-08-31T22:16:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 21041 2024-08-31T22:16:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:32.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:32.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:32.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:32.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:32.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=26, tag_tid=2, entry_tid=21138], [object_number=25, tag_tid=2, entry_tid=21137], [object_number=24, tag_tid=2, entry_tid=21136], [object_number=27, tag_tid=2, entry_tid=21135]]], state=connected] 2024-08-31T22:16:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=26, tag_tid=2, entry_tid=21138], [object_number=25, tag_tid=2, entry_tid=21137], [object_number=24, tag_tid=2, entry_tid=21136], [object_number=27, tag_tid=2, entry_tid=21135]]], state=connected] 2024-08-31T22:16:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=26, tag_tid=2, entry_tid=21138' 2024-08-31T22:16:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=26, tag_tid=2, entry_tid=21138' 2024-08-31T22:16:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=26, tag_tid=2, entry_tid=21138' '!=' 'object_number=25, tag_tid=2, entry_tid=21041' 2024-08-31T22:16:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=26, tag_tid=2, entry_tid=21138' = 'object_number=25, tag_tid=2, entry_tid=21041' 2024-08-31T22:16:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=26, tag_tid=2, entry_tid=21138' 2024-08-31T22:16:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=26, tag_tid=2, entry_tid=21138' 2024-08-31T22:16:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:33.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:33.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=26, tag_tid=2, entry_tid=21138' 2024-08-31T22:16:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21138 2024-08-31T22:16:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 21138 2024-08-31T22:16:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:33.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:33.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:33.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:33.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:33.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:33.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:33.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:33.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:33.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:33.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:33.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:33.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:33.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:33.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:33.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:33.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:33.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:33.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:33.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:33.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:33.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:33.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=21276], [object_number=27, tag_tid=2, entry_tid=21275], [object_number=26, tag_tid=2, entry_tid=21274], [object_number=25, tag_tid=2, entry_tid=21273]]], state=connected] 2024-08-31T22:16:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=21276], [object_number=27, tag_tid=2, entry_tid=21275], [object_number=26, tag_tid=2, entry_tid=21274], [object_number=25, tag_tid=2, entry_tid=21273]]], state=connected] 2024-08-31T22:16:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=24, tag_tid=2, entry_tid=21276' 2024-08-31T22:16:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=24, tag_tid=2, entry_tid=21276' 2024-08-31T22:16:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=21276' '!=' 'object_number=26, tag_tid=2, entry_tid=21138' 2024-08-31T22:16:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=21276' = 'object_number=26, tag_tid=2, entry_tid=21138' 2024-08-31T22:16:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=24, tag_tid=2, entry_tid=21276' 2024-08-31T22:16:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=21276' 2024-08-31T22:16:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=21276' 2024-08-31T22:16:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21276 2024-08-31T22:16:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 21276 2024-08-31T22:16:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:33.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=21368], [object_number=27, tag_tid=2, entry_tid=21367], [object_number=26, tag_tid=2, entry_tid=21366], [object_number=25, tag_tid=2, entry_tid=21365]]], state=connected] 2024-08-31T22:16:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:34.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:34.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:34.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:34.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:34.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=21368], [object_number=27, tag_tid=2, entry_tid=21367], [object_number=26, tag_tid=2, entry_tid=21366], [object_number=25, tag_tid=2, entry_tid=21365]]], state=connected] 2024-08-31T22:16:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=24, tag_tid=2, entry_tid=21368' 2024-08-31T22:16:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=24, tag_tid=2, entry_tid=21368' 2024-08-31T22:16:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=21368' '!=' 'object_number=24, tag_tid=2, entry_tid=21276' 2024-08-31T22:16:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=21368' = 'object_number=24, tag_tid=2, entry_tid=21276' 2024-08-31T22:16:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=24, tag_tid=2, entry_tid=21368' 2024-08-31T22:16:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=21368' 2024-08-31T22:16:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=21368' 2024-08-31T22:16:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21368 2024-08-31T22:16:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 21368 2024-08-31T22:16:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=27, tag_tid=2, entry_tid=21463], [object_number=26, tag_tid=2, entry_tid=21462], [object_number=25, tag_tid=2, entry_tid=21461], [object_number=24, tag_tid=2, entry_tid=21460]]], state=connected] 2024-08-31T22:16:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=27, tag_tid=2, entry_tid=21463], [object_number=26, tag_tid=2, entry_tid=21462], [object_number=25, tag_tid=2, entry_tid=21461], [object_number=24, tag_tid=2, entry_tid=21460]]], state=connected] 2024-08-31T22:16:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=27, tag_tid=2, entry_tid=21463' 2024-08-31T22:16:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=27, tag_tid=2, entry_tid=21463' 2024-08-31T22:16:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=27, tag_tid=2, entry_tid=21463' '!=' 'object_number=24, tag_tid=2, entry_tid=21368' 2024-08-31T22:16:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=27, tag_tid=2, entry_tid=21463' = 'object_number=24, tag_tid=2, entry_tid=21368' 2024-08-31T22:16:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=27, tag_tid=2, entry_tid=21463' 2024-08-31T22:16:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=27, tag_tid=2, entry_tid=21463' 2024-08-31T22:16:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=27, tag_tid=2, entry_tid=21463' 2024-08-31T22:16:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21463 2024-08-31T22:16:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 21463 2024-08-31T22:16:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=21565], [object_number=24, tag_tid=2, entry_tid=21564], [object_number=27, tag_tid=2, entry_tid=21563], [object_number=26, tag_tid=2, entry_tid=21562]]], state=connected] 2024-08-31T22:16:34.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=21565], [object_number=24, tag_tid=2, entry_tid=21564], [object_number=27, tag_tid=2, entry_tid=21563], [object_number=26, tag_tid=2, entry_tid=21562]]], state=connected] 2024-08-31T22:16:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=21565' 2024-08-31T22:16:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=21565' 2024-08-31T22:16:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=21565' '!=' 'object_number=27, tag_tid=2, entry_tid=21463' 2024-08-31T22:16:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=21565' = 'object_number=27, tag_tid=2, entry_tid=21463' 2024-08-31T22:16:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=25, tag_tid=2, entry_tid=21565' 2024-08-31T22:16:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=21565' 2024-08-31T22:16:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=21565' 2024-08-31T22:16:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21565 2024-08-31T22:16:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 21565 2024-08-31T22:16:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:35.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=21664], [object_number=27, tag_tid=2, entry_tid=21663], [object_number=26, tag_tid=2, entry_tid=21662], [object_number=25, tag_tid=2, entry_tid=21661]]], state=connected] 2024-08-31T22:16:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=21664], [object_number=27, tag_tid=2, entry_tid=21663], [object_number=26, tag_tid=2, entry_tid=21662], [object_number=25, tag_tid=2, entry_tid=21661]]], state=connected] 2024-08-31T22:16:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=24, tag_tid=2, entry_tid=21664' 2024-08-31T22:16:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=24, tag_tid=2, entry_tid=21664' 2024-08-31T22:16:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=21664' '!=' 'object_number=25, tag_tid=2, entry_tid=21565' 2024-08-31T22:16:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=21664' = 'object_number=25, tag_tid=2, entry_tid=21565' 2024-08-31T22:16:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=24, tag_tid=2, entry_tid=21664' 2024-08-31T22:16:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=21664' 2024-08-31T22:16:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=21664' 2024-08-31T22:16:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21664 2024-08-31T22:16:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 21664 2024-08-31T22:16:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:35.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:35.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:35.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=27, tag_tid=2, entry_tid=21783], [object_number=26, tag_tid=2, entry_tid=21782], [object_number=25, tag_tid=2, entry_tid=21781], [object_number=24, tag_tid=2, entry_tid=21780]]], state=connected] 2024-08-31T22:16:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=27, tag_tid=2, entry_tid=21783], [object_number=26, tag_tid=2, entry_tid=21782], [object_number=25, tag_tid=2, entry_tid=21781], [object_number=24, tag_tid=2, entry_tid=21780]]], state=connected] 2024-08-31T22:16:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=27, tag_tid=2, entry_tid=21783' 2024-08-31T22:16:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=27, tag_tid=2, entry_tid=21783' 2024-08-31T22:16:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=27, tag_tid=2, entry_tid=21783' '!=' 'object_number=24, tag_tid=2, entry_tid=21664' 2024-08-31T22:16:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=27, tag_tid=2, entry_tid=21783' = 'object_number=24, tag_tid=2, entry_tid=21664' 2024-08-31T22:16:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=27, tag_tid=2, entry_tid=21783' 2024-08-31T22:16:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=27, tag_tid=2, entry_tid=21783' 2024-08-31T22:16:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=27, tag_tid=2, entry_tid=21783' 2024-08-31T22:16:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21783 2024-08-31T22:16:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 21783 2024-08-31T22:16:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:36.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=21904], [object_number=27, tag_tid=2, entry_tid=21903], [object_number=26, tag_tid=2, entry_tid=21902], [object_number=25, tag_tid=2, entry_tid=21901]]], state=connected] 2024-08-31T22:16:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=21904], [object_number=27, tag_tid=2, entry_tid=21903], [object_number=26, tag_tid=2, entry_tid=21902], [object_number=25, tag_tid=2, entry_tid=21901]]], state=connected] 2024-08-31T22:16:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=24, tag_tid=2, entry_tid=21904' 2024-08-31T22:16:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=24, tag_tid=2, entry_tid=21904' 2024-08-31T22:16:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=21904' '!=' 'object_number=27, tag_tid=2, entry_tid=21783' 2024-08-31T22:16:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=21904' = 'object_number=27, tag_tid=2, entry_tid=21783' 2024-08-31T22:16:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=24, tag_tid=2, entry_tid=21904' 2024-08-31T22:16:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=21904' 2024-08-31T22:16:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=21904' 2024-08-31T22:16:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21904 2024-08-31T22:16:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 21904 2024-08-31T22:16:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:36.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=22013], [object_number=24, tag_tid=2, entry_tid=22012], [object_number=27, tag_tid=2, entry_tid=22011], [object_number=26, tag_tid=2, entry_tid=22010]]], state=connected] 2024-08-31T22:16:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=22013], [object_number=24, tag_tid=2, entry_tid=22012], [object_number=27, tag_tid=2, entry_tid=22011], [object_number=26, tag_tid=2, entry_tid=22010]]], state=connected] 2024-08-31T22:16:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=22013' 2024-08-31T22:16:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=22013' 2024-08-31T22:16:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=22013' '!=' 'object_number=24, tag_tid=2, entry_tid=21904' 2024-08-31T22:16:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=22013' = 'object_number=24, tag_tid=2, entry_tid=21904' 2024-08-31T22:16:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=25, tag_tid=2, entry_tid=22013' 2024-08-31T22:16:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=22013' 2024-08-31T22:16:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=22013' 2024-08-31T22:16:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=22013 2024-08-31T22:16:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 22013 2024-08-31T22:16:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:37.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=22117], [object_number=24, tag_tid=2, entry_tid=22116], [object_number=27, tag_tid=2, entry_tid=22115], [object_number=26, tag_tid=2, entry_tid=22114]]], state=connected] 2024-08-31T22:16:37.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=22117], [object_number=24, tag_tid=2, entry_tid=22116], [object_number=27, tag_tid=2, entry_tid=22115], [object_number=26, tag_tid=2, entry_tid=22114]]], state=connected] 2024-08-31T22:16:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=22117' 2024-08-31T22:16:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=22117' 2024-08-31T22:16:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=22117' '!=' 'object_number=25, tag_tid=2, entry_tid=22013' 2024-08-31T22:16:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=22117' = 'object_number=25, tag_tid=2, entry_tid=22013' 2024-08-31T22:16:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=25, tag_tid=2, entry_tid=22117' 2024-08-31T22:16:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=22117' 2024-08-31T22:16:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=22117' 2024-08-31T22:16:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=22117 2024-08-31T22:16:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 22117 2024-08-31T22:16:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=22221], [object_number=24, tag_tid=2, entry_tid=22220], [object_number=27, tag_tid=2, entry_tid=22219], [object_number=26, tag_tid=2, entry_tid=22218]]], state=connected] 2024-08-31T22:16:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=22221], [object_number=24, tag_tid=2, entry_tid=22220], [object_number=27, tag_tid=2, entry_tid=22219], [object_number=26, tag_tid=2, entry_tid=22218]]], state=connected] 2024-08-31T22:16:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=22221' 2024-08-31T22:16:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=22221' 2024-08-31T22:16:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=22221' '!=' 'object_number=25, tag_tid=2, entry_tid=22117' 2024-08-31T22:16:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=22221' = 'object_number=25, tag_tid=2, entry_tid=22117' 2024-08-31T22:16:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=25, tag_tid=2, entry_tid=22221' 2024-08-31T22:16:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=22221' 2024-08-31T22:16:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=22221' 2024-08-31T22:16:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=22221 2024-08-31T22:16:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 22221 2024-08-31T22:16:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:38.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=22324], [object_number=27, tag_tid=2, entry_tid=22323], [object_number=26, tag_tid=2, entry_tid=22322], [object_number=25, tag_tid=2, entry_tid=22321]]], state=connected] 2024-08-31T22:16:38.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=22324], [object_number=27, tag_tid=2, entry_tid=22323], [object_number=26, tag_tid=2, entry_tid=22322], [object_number=25, tag_tid=2, entry_tid=22321]]], state=connected] 2024-08-31T22:16:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=24, tag_tid=2, entry_tid=22324' 2024-08-31T22:16:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=24, tag_tid=2, entry_tid=22324' 2024-08-31T22:16:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=22324' '!=' 'object_number=25, tag_tid=2, entry_tid=22221' 2024-08-31T22:16:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=22324' = 'object_number=25, tag_tid=2, entry_tid=22221' 2024-08-31T22:16:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=24, tag_tid=2, entry_tid=22324' 2024-08-31T22:16:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=22324' 2024-08-31T22:16:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=22324' 2024-08-31T22:16:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=22324 2024-08-31T22:16:38.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 22324 2024-08-31T22:16:38.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:38.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:38.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:38.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:38.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:38.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:38.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:38.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=26, tag_tid=2, entry_tid=22442], [object_number=25, tag_tid=2, entry_tid=22441], [object_number=24, tag_tid=2, entry_tid=22440], [object_number=27, tag_tid=2, entry_tid=22439]]], state=connected] 2024-08-31T22:16:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:38.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:38.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:38.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:38.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:38.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=26, tag_tid=2, entry_tid=22442], [object_number=25, tag_tid=2, entry_tid=22441], [object_number=24, tag_tid=2, entry_tid=22440], [object_number=27, tag_tid=2, entry_tid=22439]]], state=connected] 2024-08-31T22:16:39.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=26, tag_tid=2, entry_tid=22442' 2024-08-31T22:16:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=26, tag_tid=2, entry_tid=22442' 2024-08-31T22:16:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=26, tag_tid=2, entry_tid=22442' '!=' 'object_number=24, tag_tid=2, entry_tid=22324' 2024-08-31T22:16:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=26, tag_tid=2, entry_tid=22442' = 'object_number=24, tag_tid=2, entry_tid=22324' 2024-08-31T22:16:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=26, tag_tid=2, entry_tid=22442' 2024-08-31T22:16:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=26, tag_tid=2, entry_tid=22442' 2024-08-31T22:16:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=26, tag_tid=2, entry_tid=22442' 2024-08-31T22:16:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=22442 2024-08-31T22:16:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 22442 2024-08-31T22:16:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:39.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=22616], [object_number=27, tag_tid=2, entry_tid=22615], [object_number=26, tag_tid=2, entry_tid=22614], [object_number=25, tag_tid=2, entry_tid=22613]]], state=connected] 2024-08-31T22:16:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=22616], [object_number=27, tag_tid=2, entry_tid=22615], [object_number=26, tag_tid=2, entry_tid=22614], [object_number=25, tag_tid=2, entry_tid=22613]]], state=connected] 2024-08-31T22:16:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=24, tag_tid=2, entry_tid=22616' 2024-08-31T22:16:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=24, tag_tid=2, entry_tid=22616' 2024-08-31T22:16:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=22616' '!=' 'object_number=26, tag_tid=2, entry_tid=22442' 2024-08-31T22:16:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=22616' = 'object_number=26, tag_tid=2, entry_tid=22442' 2024-08-31T22:16:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=24, tag_tid=2, entry_tid=22616' 2024-08-31T22:16:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=22616' 2024-08-31T22:16:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=22616' 2024-08-31T22:16:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=22616 2024-08-31T22:16:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 22616 2024-08-31T22:16:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=22685], [object_number=24, tag_tid=2, entry_tid=22684], [object_number=27, tag_tid=2, entry_tid=22683], [object_number=26, tag_tid=2, entry_tid=22682]]], state=connected] 2024-08-31T22:16:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=22685], [object_number=24, tag_tid=2, entry_tid=22684], [object_number=27, tag_tid=2, entry_tid=22683], [object_number=26, tag_tid=2, entry_tid=22682]]], state=connected] 2024-08-31T22:16:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=22685' 2024-08-31T22:16:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=22685' 2024-08-31T22:16:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=22685' '!=' 'object_number=24, tag_tid=2, entry_tid=22616' 2024-08-31T22:16:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=22685' = 'object_number=24, tag_tid=2, entry_tid=22616' 2024-08-31T22:16:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=25, tag_tid=2, entry_tid=22685' 2024-08-31T22:16:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=22685' 2024-08-31T22:16:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=22685' 2024-08-31T22:16:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=22685 2024-08-31T22:16:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 22685 2024-08-31T22:16:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=22804], [object_number=27, tag_tid=2, entry_tid=22803], [object_number=26, tag_tid=2, entry_tid=22802], [object_number=25, tag_tid=2, entry_tid=22801]]], state=connected] 2024-08-31T22:16:40.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:40.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=22804], [object_number=27, tag_tid=2, entry_tid=22803], [object_number=26, tag_tid=2, entry_tid=22802], [object_number=25, tag_tid=2, entry_tid=22801]]], state=connected] 2024-08-31T22:16:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=24, tag_tid=2, entry_tid=22804' 2024-08-31T22:16:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=24, tag_tid=2, entry_tid=22804' 2024-08-31T22:16:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=22804' '!=' 'object_number=25, tag_tid=2, entry_tid=22685' 2024-08-31T22:16:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=22804' = 'object_number=25, tag_tid=2, entry_tid=22685' 2024-08-31T22:16:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=24, tag_tid=2, entry_tid=22804' 2024-08-31T22:16:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=22804' 2024-08-31T22:16:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=22804' 2024-08-31T22:16:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=22804 2024-08-31T22:16:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 22804 2024-08-31T22:16:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:40.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:40.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:40.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=22933], [object_number=24, tag_tid=2, entry_tid=22932], [object_number=27, tag_tid=2, entry_tid=22931], [object_number=26, tag_tid=2, entry_tid=22930]]], state=connected] 2024-08-31T22:16:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=22933], [object_number=24, tag_tid=2, entry_tid=22932], [object_number=27, tag_tid=2, entry_tid=22931], [object_number=26, tag_tid=2, entry_tid=22930]]], state=connected] 2024-08-31T22:16:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=22933' 2024-08-31T22:16:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=22933' 2024-08-31T22:16:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=22933' '!=' 'object_number=24, tag_tid=2, entry_tid=22804' 2024-08-31T22:16:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=22933' = 'object_number=24, tag_tid=2, entry_tid=22804' 2024-08-31T22:16:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=25, tag_tid=2, entry_tid=22933' 2024-08-31T22:16:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=22933' 2024-08-31T22:16:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=22933' 2024-08-31T22:16:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=22933 2024-08-31T22:16:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 22933 2024-08-31T22:16:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:41.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:41.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=23052], [object_number=27, tag_tid=2, entry_tid=23051], [object_number=26, tag_tid=2, entry_tid=23050], [object_number=25, tag_tid=2, entry_tid=23049]]], state=connected] 2024-08-31T22:16:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:41.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:41.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:41.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:41.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:41.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:41.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=23052], [object_number=27, tag_tid=2, entry_tid=23051], [object_number=26, tag_tid=2, entry_tid=23050], [object_number=25, tag_tid=2, entry_tid=23049]]], state=connected] 2024-08-31T22:16:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=24, tag_tid=2, entry_tid=23052' 2024-08-31T22:16:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=24, tag_tid=2, entry_tid=23052' 2024-08-31T22:16:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=23052' '!=' 'object_number=25, tag_tid=2, entry_tid=22933' 2024-08-31T22:16:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=23052' = 'object_number=25, tag_tid=2, entry_tid=22933' 2024-08-31T22:16:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=24, tag_tid=2, entry_tid=23052' 2024-08-31T22:16:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=23052' 2024-08-31T22:16:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=23052' 2024-08-31T22:16:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=23052 2024-08-31T22:16:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 23052 2024-08-31T22:16:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=23169], [object_number=24, tag_tid=2, entry_tid=23168], [object_number=27, tag_tid=2, entry_tid=23167], [object_number=26, tag_tid=2, entry_tid=23166]]], state=connected] 2024-08-31T22:16:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:41.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:41.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:41.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=23169], [object_number=24, tag_tid=2, entry_tid=23168], [object_number=27, tag_tid=2, entry_tid=23167], [object_number=26, tag_tid=2, entry_tid=23166]]], state=connected] 2024-08-31T22:16:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=23169' 2024-08-31T22:16:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=23169' 2024-08-31T22:16:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=23169' '!=' 'object_number=24, tag_tid=2, entry_tid=23052' 2024-08-31T22:16:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=23169' = 'object_number=24, tag_tid=2, entry_tid=23052' 2024-08-31T22:16:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=25, tag_tid=2, entry_tid=23169' 2024-08-31T22:16:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=23169' 2024-08-31T22:16:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=23169' 2024-08-31T22:16:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=23169 2024-08-31T22:16:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 23169 2024-08-31T22:16:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:42.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=23285], [object_number=24, tag_tid=2, entry_tid=23284], [object_number=27, tag_tid=2, entry_tid=23283], [object_number=26, tag_tid=2, entry_tid=23282]]], state=connected] 2024-08-31T22:16:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=23285], [object_number=24, tag_tid=2, entry_tid=23284], [object_number=27, tag_tid=2, entry_tid=23283], [object_number=26, tag_tid=2, entry_tid=23282]]], state=connected] 2024-08-31T22:16:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=23285' 2024-08-31T22:16:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=23285' 2024-08-31T22:16:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=23285' '!=' 'object_number=25, tag_tid=2, entry_tid=23169' 2024-08-31T22:16:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=23285' = 'object_number=25, tag_tid=2, entry_tid=23169' 2024-08-31T22:16:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=25, tag_tid=2, entry_tid=23285' 2024-08-31T22:16:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=23285' 2024-08-31T22:16:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=23285' 2024-08-31T22:16:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=23285 2024-08-31T22:16:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 23285 2024-08-31T22:16:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=23397], [object_number=24, tag_tid=2, entry_tid=23396], [object_number=27, tag_tid=2, entry_tid=23395], [object_number=26, tag_tid=2, entry_tid=23394]]], state=connected] 2024-08-31T22:16:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:42.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:42.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=23397], [object_number=24, tag_tid=2, entry_tid=23396], [object_number=27, tag_tid=2, entry_tid=23395], [object_number=26, tag_tid=2, entry_tid=23394]]], state=connected] 2024-08-31T22:16:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=23397' 2024-08-31T22:16:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=23397' 2024-08-31T22:16:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=23397' '!=' 'object_number=25, tag_tid=2, entry_tid=23285' 2024-08-31T22:16:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=23397' = 'object_number=25, tag_tid=2, entry_tid=23285' 2024-08-31T22:16:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=25, tag_tid=2, entry_tid=23397' 2024-08-31T22:16:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=23397' 2024-08-31T22:16:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=23397' 2024-08-31T22:16:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=23397 2024-08-31T22:16:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 23397 2024-08-31T22:16:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:43.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:43.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:43.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:43.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:43.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:43.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:43.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:43.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:43.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:43.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:43.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:43.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:43.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:43.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:43.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:43.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:43.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506], [object_number=25, tag_tid=2, entry_tid=23505], [object_number=24, tag_tid=2, entry_tid=23504]]], state=connected] 2024-08-31T22:16:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506], [object_number=25, tag_tid=2, entry_tid=23505], [object_number=24, tag_tid=2, entry_tid=23504]]], state=connected] 2024-08-31T22:16:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=27, tag_tid=2, entry_tid=23507' 2024-08-31T22:16:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=27, tag_tid=2, entry_tid=23507' 2024-08-31T22:16:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=27, tag_tid=2, entry_tid=23507' '!=' 'object_number=25, tag_tid=2, entry_tid=23397' 2024-08-31T22:16:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:16:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=27, tag_tid=2, entry_tid=23507' = 'object_number=25, tag_tid=2, entry_tid=23397' 2024-08-31T22:16:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=27, tag_tid=2, entry_tid=23507' 2024-08-31T22:16:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:16:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=27, tag_tid=2, entry_tid=23507' 2024-08-31T22:16:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:16:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:16:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:16:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=27, tag_tid=2, entry_tid=23507' 2024-08-31T22:16:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:16:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:16:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=23507 2024-08-31T22:16:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 23507 2024-08-31T22:16:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:16:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:16:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:16:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:16:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:16:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:16:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:16:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:16:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:16:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:16:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:16:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:16:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:16:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:16:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:16:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:16:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:16:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:16:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:16:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:16:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:16:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:16:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:16:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:16:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:16:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:16:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:16:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:16:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:16:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:16:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:16:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:16:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:16:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:16:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:16:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:16:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:16:43.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:16:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:16:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:16:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:16:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:16:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:16:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:16:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:16:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:16:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:16:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:16:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:16:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:16:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 6 2024-08-31T22:16:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 6 2024-08-31T22:16:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:16:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=23507], [object_number=26, tag_tid=2, entry_tid=23506]]], state=connected] 2024-08-31T22:16:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:16:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=3, entry_tid=1' 2024-08-31T22:16:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-08-31T22:16:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap1 2024-08-31T22:16:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:16:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:16:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:16:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-08-31T22:16:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'test': 2024-08-31T22:16:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:16:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:16:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:16:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a87b4c6a1b 2024-08-31T22:16:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a87b4c6a1b 2024-08-31T22:16:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:16:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:16:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:16:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:16:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:14:57 2024 2024-08-31T22:16:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:14:57 2024 2024-08-31T22:16:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:15:58 2024 2024-08-31T22:16:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:16:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a87b4c6a1b 2024-08-31T22:16:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:16:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:16:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:16:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:16:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:16:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap1 2024-08-31T22:16:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap1 2024-08-31T22:16:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:16:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:16:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:16:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:16:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:16:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:16:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:16:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:16:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:16:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:16:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ clean_snap_name=snap1-clean 2024-08-31T22:16:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-08-31T22:16:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:16:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap1-clean 2024-08-31T22:16:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-08-31T22:16:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:16:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror test 2024-08-31T22:16:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:16:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:16:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=10 2024-08-31T22:16:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:16:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-08-31T22:16:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 48101376 bytes 2024-08-31T22:16:51.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 81985536 bytes 2024-08-31T22:16:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 113642496 bytes 2024-08-31T22:16:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 185755648 bytes 2024-08-31T22:16:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:500 IOs, 251487232 bytes 2024-08-31T22:16:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:600 IOs, 286859264 bytes 2024-08-31T22:16:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:16:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:16:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:16:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-08-31T22:16:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:16:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:16:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:16:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:16:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:16:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-08-31T22:16:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:16:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:16:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:16:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:16:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:16:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-08-31T22:16:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='test: 2024-08-31T22:16:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: global_id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+replaying 2024-08-31T22:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: replaying, {"bytes_per_second":2911243.6,"entries_behind_primary":1,"entries_per_second":181.07,"non_primary_position":{"entry_tid":1,"object_number":25,"tag_tid":3},"primary_position":{"entry_tid":1,"object_number":25,"tag_tid":4},"seconds_until_synced":0} 2024-08-31T22:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: service: mirror.0 on smithi032 2024-08-31T22:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:16:55 2024-08-31T22:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: peer_sites: 2024-08-31T22:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: name: cluster2 2024-08-31T22:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+stopped 2024-08-31T22:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: local image is primary 2024-08-31T22:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:16:55' 2024-08-31T22:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ state: .*up+replaying' 2024-08-31T22:16:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: state: up+replaying 2024-08-31T22:16:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ description: .*' 2024-08-31T22:16:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: description: replaying, {"bytes_per_second":2911243.6,"entries_behind_primary":1,"entries_per_second":181.07,"non_primary_position":{"entry_tid":1,"object_number":25,"tag_tid":3},"primary_position":{"entry_tid":1,"object_number":25,"tag_tid":4},"seconds_until_synced":0} 2024-08-31T22:16:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:16:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:16:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^up+' 2024-08-31T22:16:58.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:16:58.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep 'service: *mirror..* on ' 2024-08-31T22:16:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: service: mirror.0 on smithi032 2024-08-31T22:16:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update 2024-08-31T22:16:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-08-31T22:16:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:16:55' 2024-08-31T22:16:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-08-31 22:16:55' 2024-08-31T22:16:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:16:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:16:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'prev_last_update=2024-08-31 22:16:55' 2024-08-31T22:16:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:16:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:16:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-08-31T22:16:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='OKOKOK142574264mirror.0smithi03219.1.1-244-g3091dfc0trueOK42614267mirror.1smithi03219.1.1-244-g3091dfc0falseOKtestdd2c705e-25cf-4565-981f-d48de48eee84up+replayingreplaying, {"bytes_per_second":2911243.6,"entries_behind_primary":1,"entries_per_second":181.07,"non_primary_position":{"entry_tid":1,"object_number":25,"tag_tid":3},"primary_position":{"entry_tid":1,"object_number":25,"tag_tid":4},"seconds_until_synced":0}42574264mirror.0smithi0322024-08-31 22:16:55cluster24ca32b00-2164-486e-837e-e93795a16af1up+stoppedlocal image is primary2024-08-31 22:16:55' 2024-08-31T22:16:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update state 2024-08-31T22:16:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-08-31T22:16:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:16:55' 2024-08-31T22:16:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-08-31T22:16:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ state=up+replaying 2024-08-31T22:16:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:16:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep up+replaying 2024-08-31T22:16:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:16:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:16:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap2 2024-08-31T22:16:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap2 2024-08-31T22:16:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:16:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:16:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:16:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap2 2024-08-31T22:16:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:17:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-08-31T22:17:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-08-31T22:17:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:17:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:17:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:17:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror test started 2024-08-31T22:17:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:17:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:17:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:17:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-08-31T22:17:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:17:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:17:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:17:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:17:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:17:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:17:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:17:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-08-31T22:17:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:17:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:17:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:17:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:17:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:17:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:17:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:17:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:17:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:17:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:17:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:17:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:17:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:17:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:17:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:17:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:17:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-08-31T22:17:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:17:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:17:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:17:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:17:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:17:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:17:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:17:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:17:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:17:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:17:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:17:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:17:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:17:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:17:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:17:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:17:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:32.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:32.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:32.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:32.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:32.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:32.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:32.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:32.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:32.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:32.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:32.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:32.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:32.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=8405], [object_number=36, tag_tid=5, entry_tid=8404], [object_number=39, tag_tid=5, entry_tid=8403], [object_number=38, tag_tid=5, entry_tid=8402]]], state=connected] 2024-08-31T22:17:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=8405], [object_number=36, tag_tid=5, entry_tid=8404], [object_number=39, tag_tid=5, entry_tid=8403], [object_number=38, tag_tid=5, entry_tid=8402]]], state=connected] 2024-08-31T22:17:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=8405' 2024-08-31T22:17:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=8405' 2024-08-31T22:17:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=8405' '!=' '' 2024-08-31T22:17:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=8405' = '' 2024-08-31T22:17:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=8405' 2024-08-31T22:17:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=8405' 2024-08-31T22:17:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=8405' 2024-08-31T22:17:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8405 2024-08-31T22:17:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 8405 2024-08-31T22:17:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=8522], [object_number=37, tag_tid=5, entry_tid=8521], [object_number=36, tag_tid=5, entry_tid=8520], [object_number=39, tag_tid=5, entry_tid=8519]]], state=connected] 2024-08-31T22:17:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=8522], [object_number=37, tag_tid=5, entry_tid=8521], [object_number=36, tag_tid=5, entry_tid=8520], [object_number=39, tag_tid=5, entry_tid=8519]]], state=connected] 2024-08-31T22:17:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=38, tag_tid=5, entry_tid=8522' 2024-08-31T22:17:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=38, tag_tid=5, entry_tid=8522' 2024-08-31T22:17:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=8522' '!=' 'object_number=37, tag_tid=5, entry_tid=8405' 2024-08-31T22:17:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=8522' = 'object_number=37, tag_tid=5, entry_tid=8405' 2024-08-31T22:17:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=38, tag_tid=5, entry_tid=8522' 2024-08-31T22:17:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=8522' 2024-08-31T22:17:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=8522' 2024-08-31T22:17:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8522 2024-08-31T22:17:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 8522 2024-08-31T22:17:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=8633], [object_number=36, tag_tid=5, entry_tid=8632], [object_number=39, tag_tid=5, entry_tid=8631], [object_number=38, tag_tid=5, entry_tid=8630]]], state=connected] 2024-08-31T22:17:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:33.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:33.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:33.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:33.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:33.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:33.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=8633], [object_number=36, tag_tid=5, entry_tid=8632], [object_number=39, tag_tid=5, entry_tid=8631], [object_number=38, tag_tid=5, entry_tid=8630]]], state=connected] 2024-08-31T22:17:33.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=8633' 2024-08-31T22:17:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=8633' 2024-08-31T22:17:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=8633' '!=' 'object_number=38, tag_tid=5, entry_tid=8522' 2024-08-31T22:17:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=8633' = 'object_number=38, tag_tid=5, entry_tid=8522' 2024-08-31T22:17:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=8633' 2024-08-31T22:17:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=8633' 2024-08-31T22:17:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=8633' 2024-08-31T22:17:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8633 2024-08-31T22:17:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 8633 2024-08-31T22:17:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:33.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:33.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:33.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:33.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:33.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:33.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:33.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:33.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=8747], [object_number=38, tag_tid=5, entry_tid=8746], [object_number=37, tag_tid=5, entry_tid=8745], [object_number=36, tag_tid=5, entry_tid=8744]]], state=connected] 2024-08-31T22:17:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:33.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:33.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=8747], [object_number=38, tag_tid=5, entry_tid=8746], [object_number=37, tag_tid=5, entry_tid=8745], [object_number=36, tag_tid=5, entry_tid=8744]]], state=connected] 2024-08-31T22:17:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=39, tag_tid=5, entry_tid=8747' 2024-08-31T22:17:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=39, tag_tid=5, entry_tid=8747' 2024-08-31T22:17:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=8747' '!=' 'object_number=37, tag_tid=5, entry_tid=8633' 2024-08-31T22:17:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=8747' = 'object_number=37, tag_tid=5, entry_tid=8633' 2024-08-31T22:17:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=39, tag_tid=5, entry_tid=8747' 2024-08-31T22:17:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=8747' 2024-08-31T22:17:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:33.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=8747' 2024-08-31T22:17:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:33.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8747 2024-08-31T22:17:33.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 8747 2024-08-31T22:17:33.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=8859], [object_number=38, tag_tid=5, entry_tid=8858], [object_number=37, tag_tid=5, entry_tid=8857], [object_number=36, tag_tid=5, entry_tid=8856]]], state=connected] 2024-08-31T22:17:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=8859], [object_number=38, tag_tid=5, entry_tid=8858], [object_number=37, tag_tid=5, entry_tid=8857], [object_number=36, tag_tid=5, entry_tid=8856]]], state=connected] 2024-08-31T22:17:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=39, tag_tid=5, entry_tid=8859' 2024-08-31T22:17:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=39, tag_tid=5, entry_tid=8859' 2024-08-31T22:17:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=8859' '!=' 'object_number=39, tag_tid=5, entry_tid=8747' 2024-08-31T22:17:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=8859' = 'object_number=39, tag_tid=5, entry_tid=8747' 2024-08-31T22:17:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=39, tag_tid=5, entry_tid=8859' 2024-08-31T22:17:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:34.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=8859' 2024-08-31T22:17:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=8859' 2024-08-31T22:17:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8859 2024-08-31T22:17:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 8859 2024-08-31T22:17:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:34.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=8993], [object_number=36, tag_tid=5, entry_tid=8992], [object_number=39, tag_tid=5, entry_tid=8991], [object_number=38, tag_tid=5, entry_tid=8990]]], state=connected] 2024-08-31T22:17:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:34.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:34.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=8993], [object_number=36, tag_tid=5, entry_tid=8992], [object_number=39, tag_tid=5, entry_tid=8991], [object_number=38, tag_tid=5, entry_tid=8990]]], state=connected] 2024-08-31T22:17:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=8993' 2024-08-31T22:17:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=8993' 2024-08-31T22:17:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=8993' '!=' 'object_number=39, tag_tid=5, entry_tid=8859' 2024-08-31T22:17:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=8993' = 'object_number=39, tag_tid=5, entry_tid=8859' 2024-08-31T22:17:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=8993' 2024-08-31T22:17:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=8993' 2024-08-31T22:17:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=8993' 2024-08-31T22:17:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8993 2024-08-31T22:17:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 8993 2024-08-31T22:17:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:35.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:35.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=9112], [object_number=39, tag_tid=5, entry_tid=9111], [object_number=38, tag_tid=5, entry_tid=9110], [object_number=37, tag_tid=5, entry_tid=9109]]], state=connected] 2024-08-31T22:17:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:35.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:35.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:35.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:35.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:35.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=9112], [object_number=39, tag_tid=5, entry_tid=9111], [object_number=38, tag_tid=5, entry_tid=9110], [object_number=37, tag_tid=5, entry_tid=9109]]], state=connected] 2024-08-31T22:17:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=36, tag_tid=5, entry_tid=9112' 2024-08-31T22:17:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=36, tag_tid=5, entry_tid=9112' 2024-08-31T22:17:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=9112' '!=' 'object_number=37, tag_tid=5, entry_tid=8993' 2024-08-31T22:17:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=9112' = 'object_number=37, tag_tid=5, entry_tid=8993' 2024-08-31T22:17:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=36, tag_tid=5, entry_tid=9112' 2024-08-31T22:17:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=9112' 2024-08-31T22:17:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:35.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:35.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:35.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=9112' 2024-08-31T22:17:35.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:35.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9112 2024-08-31T22:17:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 9112 2024-08-31T22:17:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:35.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:35.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:35.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:35.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=9212], [object_number=39, tag_tid=5, entry_tid=9211], [object_number=38, tag_tid=5, entry_tid=9210], [object_number=37, tag_tid=5, entry_tid=9209]]], state=connected] 2024-08-31T22:17:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=9212], [object_number=39, tag_tid=5, entry_tid=9211], [object_number=38, tag_tid=5, entry_tid=9210], [object_number=37, tag_tid=5, entry_tid=9209]]], state=connected] 2024-08-31T22:17:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=36, tag_tid=5, entry_tid=9212' 2024-08-31T22:17:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=36, tag_tid=5, entry_tid=9212' 2024-08-31T22:17:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=9212' '!=' 'object_number=36, tag_tid=5, entry_tid=9112' 2024-08-31T22:17:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=9212' = 'object_number=36, tag_tid=5, entry_tid=9112' 2024-08-31T22:17:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=36, tag_tid=5, entry_tid=9212' 2024-08-31T22:17:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=9212' 2024-08-31T22:17:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=9212' 2024-08-31T22:17:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9212 2024-08-31T22:17:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 9212 2024-08-31T22:17:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=9307], [object_number=38, tag_tid=5, entry_tid=9306], [object_number=37, tag_tid=5, entry_tid=9305], [object_number=36, tag_tid=5, entry_tid=9304]]], state=connected] 2024-08-31T22:17:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:36.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:36.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:36.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:36.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=9307], [object_number=38, tag_tid=5, entry_tid=9306], [object_number=37, tag_tid=5, entry_tid=9305], [object_number=36, tag_tid=5, entry_tid=9304]]], state=connected] 2024-08-31T22:17:36.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:36.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=39, tag_tid=5, entry_tid=9307' 2024-08-31T22:17:36.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=39, tag_tid=5, entry_tid=9307' 2024-08-31T22:17:36.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=9307' '!=' 'object_number=36, tag_tid=5, entry_tid=9212' 2024-08-31T22:17:36.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:36.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=9307' = 'object_number=36, tag_tid=5, entry_tid=9212' 2024-08-31T22:17:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=39, tag_tid=5, entry_tid=9307' 2024-08-31T22:17:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=9307' 2024-08-31T22:17:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:36.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:36.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:36.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=9307' 2024-08-31T22:17:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9307 2024-08-31T22:17:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 9307 2024-08-31T22:17:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=9401], [object_number=36, tag_tid=5, entry_tid=9400], [object_number=39, tag_tid=5, entry_tid=9399], [object_number=38, tag_tid=5, entry_tid=9398]]], state=connected] 2024-08-31T22:17:36.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=9401], [object_number=36, tag_tid=5, entry_tid=9400], [object_number=39, tag_tid=5, entry_tid=9399], [object_number=38, tag_tid=5, entry_tid=9398]]], state=connected] 2024-08-31T22:17:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=9401' 2024-08-31T22:17:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=9401' 2024-08-31T22:17:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=9401' '!=' 'object_number=39, tag_tid=5, entry_tid=9307' 2024-08-31T22:17:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=9401' = 'object_number=39, tag_tid=5, entry_tid=9307' 2024-08-31T22:17:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=9401' 2024-08-31T22:17:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=9401' 2024-08-31T22:17:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=9401' 2024-08-31T22:17:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9401 2024-08-31T22:17:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 9401 2024-08-31T22:17:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=9504], [object_number=39, tag_tid=5, entry_tid=9503], [object_number=38, tag_tid=5, entry_tid=9502], [object_number=37, tag_tid=5, entry_tid=9501]]], state=connected] 2024-08-31T22:17:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=9504], [object_number=39, tag_tid=5, entry_tid=9503], [object_number=38, tag_tid=5, entry_tid=9502], [object_number=37, tag_tid=5, entry_tid=9501]]], state=connected] 2024-08-31T22:17:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=36, tag_tid=5, entry_tid=9504' 2024-08-31T22:17:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=36, tag_tid=5, entry_tid=9504' 2024-08-31T22:17:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=9504' '!=' 'object_number=37, tag_tid=5, entry_tid=9401' 2024-08-31T22:17:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=9504' = 'object_number=37, tag_tid=5, entry_tid=9401' 2024-08-31T22:17:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=36, tag_tid=5, entry_tid=9504' 2024-08-31T22:17:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=9504' 2024-08-31T22:17:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=9504' 2024-08-31T22:17:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9504 2024-08-31T22:17:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 9504 2024-08-31T22:17:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=9607], [object_number=38, tag_tid=5, entry_tid=9606], [object_number=37, tag_tid=5, entry_tid=9605], [object_number=36, tag_tid=5, entry_tid=9604]]], state=connected] 2024-08-31T22:17:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=9607], [object_number=38, tag_tid=5, entry_tid=9606], [object_number=37, tag_tid=5, entry_tid=9605], [object_number=36, tag_tid=5, entry_tid=9604]]], state=connected] 2024-08-31T22:17:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=39, tag_tid=5, entry_tid=9607' 2024-08-31T22:17:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=39, tag_tid=5, entry_tid=9607' 2024-08-31T22:17:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=9607' '!=' 'object_number=36, tag_tid=5, entry_tid=9504' 2024-08-31T22:17:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=9607' = 'object_number=36, tag_tid=5, entry_tid=9504' 2024-08-31T22:17:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=39, tag_tid=5, entry_tid=9607' 2024-08-31T22:17:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=9607' 2024-08-31T22:17:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=9607' 2024-08-31T22:17:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9607 2024-08-31T22:17:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 9607 2024-08-31T22:17:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=9725], [object_number=36, tag_tid=5, entry_tid=9724], [object_number=39, tag_tid=5, entry_tid=9723], [object_number=38, tag_tid=5, entry_tid=9722]]], state=connected] 2024-08-31T22:17:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=9725], [object_number=36, tag_tid=5, entry_tid=9724], [object_number=39, tag_tid=5, entry_tid=9723], [object_number=38, tag_tid=5, entry_tid=9722]]], state=connected] 2024-08-31T22:17:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=9725' 2024-08-31T22:17:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=9725' 2024-08-31T22:17:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=9725' '!=' 'object_number=39, tag_tid=5, entry_tid=9607' 2024-08-31T22:17:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=9725' = 'object_number=39, tag_tid=5, entry_tid=9607' 2024-08-31T22:17:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=9725' 2024-08-31T22:17:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=9725' 2024-08-31T22:17:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:38.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:38.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=9725' 2024-08-31T22:17:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9725 2024-08-31T22:17:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 9725 2024-08-31T22:17:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:38.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:38.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:38.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:38.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:38.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:38.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=9850], [object_number=37, tag_tid=5, entry_tid=9849], [object_number=36, tag_tid=5, entry_tid=9848], [object_number=39, tag_tid=5, entry_tid=9847]]], state=connected] 2024-08-31T22:17:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=9850], [object_number=37, tag_tid=5, entry_tid=9849], [object_number=36, tag_tid=5, entry_tid=9848], [object_number=39, tag_tid=5, entry_tid=9847]]], state=connected] 2024-08-31T22:17:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=38, tag_tid=5, entry_tid=9850' 2024-08-31T22:17:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=38, tag_tid=5, entry_tid=9850' 2024-08-31T22:17:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=9850' '!=' 'object_number=37, tag_tid=5, entry_tid=9725' 2024-08-31T22:17:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=9850' = 'object_number=37, tag_tid=5, entry_tid=9725' 2024-08-31T22:17:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=38, tag_tid=5, entry_tid=9850' 2024-08-31T22:17:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=9850' 2024-08-31T22:17:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:38.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:38.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:38.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:38.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:38.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=9850' 2024-08-31T22:17:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9850 2024-08-31T22:17:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 9850 2024-08-31T22:17:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:39.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:39.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:39.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=9973], [object_number=36, tag_tid=5, entry_tid=9972], [object_number=39, tag_tid=5, entry_tid=9971], [object_number=38, tag_tid=5, entry_tid=9970]]], state=connected] 2024-08-31T22:17:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=9973], [object_number=36, tag_tid=5, entry_tid=9972], [object_number=39, tag_tid=5, entry_tid=9971], [object_number=38, tag_tid=5, entry_tid=9970]]], state=connected] 2024-08-31T22:17:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=9973' 2024-08-31T22:17:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=9973' 2024-08-31T22:17:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=9973' '!=' 'object_number=38, tag_tid=5, entry_tid=9850' 2024-08-31T22:17:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=9973' = 'object_number=38, tag_tid=5, entry_tid=9850' 2024-08-31T22:17:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=9973' 2024-08-31T22:17:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:39.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:39.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:39.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:39.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=9973' 2024-08-31T22:17:39.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:39.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=9973' 2024-08-31T22:17:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9973 2024-08-31T22:17:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 9973 2024-08-31T22:17:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:39.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:39.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:39.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:39.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:39.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:39.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:39.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:39.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:39.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=10093], [object_number=36, tag_tid=5, entry_tid=10092], [object_number=39, tag_tid=5, entry_tid=10091], [object_number=38, tag_tid=5, entry_tid=10090]]], state=connected] 2024-08-31T22:17:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:39.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:39.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=10093], [object_number=36, tag_tid=5, entry_tid=10092], [object_number=39, tag_tid=5, entry_tid=10091], [object_number=38, tag_tid=5, entry_tid=10090]]], state=connected] 2024-08-31T22:17:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=10093' 2024-08-31T22:17:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=10093' 2024-08-31T22:17:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=10093' '!=' 'object_number=37, tag_tid=5, entry_tid=9973' 2024-08-31T22:17:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=10093' = 'object_number=37, tag_tid=5, entry_tid=9973' 2024-08-31T22:17:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=10093' 2024-08-31T22:17:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=10093' 2024-08-31T22:17:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:39.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=10093' 2024-08-31T22:17:39.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:39.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10093 2024-08-31T22:17:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 10093 2024-08-31T22:17:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=10206], [object_number=37, tag_tid=5, entry_tid=10205], [object_number=36, tag_tid=5, entry_tid=10204], [object_number=39, tag_tid=5, entry_tid=10203]]], state=connected] 2024-08-31T22:17:40.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:40.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:40.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:40.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:40.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:40.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=10206], [object_number=37, tag_tid=5, entry_tid=10205], [object_number=36, tag_tid=5, entry_tid=10204], [object_number=39, tag_tid=5, entry_tid=10203]]], state=connected] 2024-08-31T22:17:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=38, tag_tid=5, entry_tid=10206' 2024-08-31T22:17:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=38, tag_tid=5, entry_tid=10206' 2024-08-31T22:17:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=10206' '!=' 'object_number=37, tag_tid=5, entry_tid=10093' 2024-08-31T22:17:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=10206' = 'object_number=37, tag_tid=5, entry_tid=10093' 2024-08-31T22:17:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=38, tag_tid=5, entry_tid=10206' 2024-08-31T22:17:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=10206' 2024-08-31T22:17:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=10206' 2024-08-31T22:17:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10206 2024-08-31T22:17:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 10206 2024-08-31T22:17:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=10310], [object_number=37, tag_tid=5, entry_tid=10309], [object_number=36, tag_tid=5, entry_tid=10308], [object_number=39, tag_tid=5, entry_tid=10307]]], state=connected] 2024-08-31T22:17:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 9 2024-08-31T22:17:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=10310], [object_number=37, tag_tid=5, entry_tid=10309], [object_number=36, tag_tid=5, entry_tid=10308], [object_number=39, tag_tid=5, entry_tid=10307]]], state=connected] 2024-08-31T22:17:40.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=38, tag_tid=5, entry_tid=10310' 2024-08-31T22:17:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=38, tag_tid=5, entry_tid=10310' 2024-08-31T22:17:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=10310' '!=' 'object_number=38, tag_tid=5, entry_tid=10206' 2024-08-31T22:17:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=10310' = 'object_number=38, tag_tid=5, entry_tid=10206' 2024-08-31T22:17:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=38, tag_tid=5, entry_tid=10310' 2024-08-31T22:17:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=10310' 2024-08-31T22:17:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:40.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:40.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=10310' 2024-08-31T22:17:40.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:40.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10310 2024-08-31T22:17:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 10310 2024-08-31T22:17:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:40.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:40.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:40.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=10358], [object_number=41, tag_tid=5, entry_tid=10357], [object_number=40, tag_tid=5, entry_tid=10356], [object_number=43, tag_tid=5, entry_tid=10355]]], state=connected] 2024-08-31T22:17:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=10358], [object_number=41, tag_tid=5, entry_tid=10357], [object_number=40, tag_tid=5, entry_tid=10356], [object_number=43, tag_tid=5, entry_tid=10355]]], state=connected] 2024-08-31T22:17:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=42, tag_tid=5, entry_tid=10358' 2024-08-31T22:17:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=42, tag_tid=5, entry_tid=10358' 2024-08-31T22:17:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=10358' '!=' 'object_number=38, tag_tid=5, entry_tid=10310' 2024-08-31T22:17:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=10358' = 'object_number=38, tag_tid=5, entry_tid=10310' 2024-08-31T22:17:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=42, tag_tid=5, entry_tid=10358' 2024-08-31T22:17:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=10358' 2024-08-31T22:17:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=10358' 2024-08-31T22:17:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10358 2024-08-31T22:17:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 10358 2024-08-31T22:17:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=10467], [object_number=42, tag_tid=5, entry_tid=10466], [object_number=41, tag_tid=5, entry_tid=10465], [object_number=40, tag_tid=5, entry_tid=10464]]], state=connected] 2024-08-31T22:17:41.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=10467], [object_number=42, tag_tid=5, entry_tid=10466], [object_number=41, tag_tid=5, entry_tid=10465], [object_number=40, tag_tid=5, entry_tid=10464]]], state=connected] 2024-08-31T22:17:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=43, tag_tid=5, entry_tid=10467' 2024-08-31T22:17:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=43, tag_tid=5, entry_tid=10467' 2024-08-31T22:17:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=10467' '!=' 'object_number=42, tag_tid=5, entry_tid=10358' 2024-08-31T22:17:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=10467' = 'object_number=42, tag_tid=5, entry_tid=10358' 2024-08-31T22:17:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=43, tag_tid=5, entry_tid=10467' 2024-08-31T22:17:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=10467' 2024-08-31T22:17:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=10467' 2024-08-31T22:17:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10467 2024-08-31T22:17:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 10467 2024-08-31T22:17:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=10570], [object_number=41, tag_tid=5, entry_tid=10569], [object_number=40, tag_tid=5, entry_tid=10568], [object_number=43, tag_tid=5, entry_tid=10567]]], state=connected] 2024-08-31T22:17:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=10570], [object_number=41, tag_tid=5, entry_tid=10569], [object_number=40, tag_tid=5, entry_tid=10568], [object_number=43, tag_tid=5, entry_tid=10567]]], state=connected] 2024-08-31T22:17:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=42, tag_tid=5, entry_tid=10570' 2024-08-31T22:17:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=42, tag_tid=5, entry_tid=10570' 2024-08-31T22:17:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=10570' '!=' 'object_number=43, tag_tid=5, entry_tid=10467' 2024-08-31T22:17:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=10570' = 'object_number=43, tag_tid=5, entry_tid=10467' 2024-08-31T22:17:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=42, tag_tid=5, entry_tid=10570' 2024-08-31T22:17:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=10570' 2024-08-31T22:17:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=10570' 2024-08-31T22:17:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10570 2024-08-31T22:17:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 10570 2024-08-31T22:17:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:42.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:42.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:42.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:42.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:42.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:42.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:42.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:42.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:42.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:42.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=10673], [object_number=40, tag_tid=5, entry_tid=10672], [object_number=43, tag_tid=5, entry_tid=10671], [object_number=42, tag_tid=5, entry_tid=10670]]], state=connected] 2024-08-31T22:17:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=10673], [object_number=40, tag_tid=5, entry_tid=10672], [object_number=43, tag_tid=5, entry_tid=10671], [object_number=42, tag_tid=5, entry_tid=10670]]], state=connected] 2024-08-31T22:17:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=10673' 2024-08-31T22:17:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=10673' 2024-08-31T22:17:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=10673' '!=' 'object_number=42, tag_tid=5, entry_tid=10570' 2024-08-31T22:17:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=10673' = 'object_number=42, tag_tid=5, entry_tid=10570' 2024-08-31T22:17:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=10673' 2024-08-31T22:17:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=10673' 2024-08-31T22:17:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=10673' 2024-08-31T22:17:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10673 2024-08-31T22:17:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 10673 2024-08-31T22:17:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=10766], [object_number=41, tag_tid=5, entry_tid=10765], [object_number=40, tag_tid=5, entry_tid=10764], [object_number=43, tag_tid=5, entry_tid=10763]]], state=connected] 2024-08-31T22:17:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=10766], [object_number=41, tag_tid=5, entry_tid=10765], [object_number=40, tag_tid=5, entry_tid=10764], [object_number=43, tag_tid=5, entry_tid=10763]]], state=connected] 2024-08-31T22:17:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=42, tag_tid=5, entry_tid=10766' 2024-08-31T22:17:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=42, tag_tid=5, entry_tid=10766' 2024-08-31T22:17:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=10766' '!=' 'object_number=41, tag_tid=5, entry_tid=10673' 2024-08-31T22:17:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=10766' = 'object_number=41, tag_tid=5, entry_tid=10673' 2024-08-31T22:17:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=42, tag_tid=5, entry_tid=10766' 2024-08-31T22:17:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=10766' 2024-08-31T22:17:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=10766' 2024-08-31T22:17:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10766 2024-08-31T22:17:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 10766 2024-08-31T22:17:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=10862], [object_number=41, tag_tid=5, entry_tid=10861], [object_number=40, tag_tid=5, entry_tid=10860], [object_number=43, tag_tid=5, entry_tid=10859]]], state=connected] 2024-08-31T22:17:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=10862], [object_number=41, tag_tid=5, entry_tid=10861], [object_number=40, tag_tid=5, entry_tid=10860], [object_number=43, tag_tid=5, entry_tid=10859]]], state=connected] 2024-08-31T22:17:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=42, tag_tid=5, entry_tid=10862' 2024-08-31T22:17:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=42, tag_tid=5, entry_tid=10862' 2024-08-31T22:17:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=10862' '!=' 'object_number=42, tag_tid=5, entry_tid=10766' 2024-08-31T22:17:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=10862' = 'object_number=42, tag_tid=5, entry_tid=10766' 2024-08-31T22:17:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=42, tag_tid=5, entry_tid=10862' 2024-08-31T22:17:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=10862' 2024-08-31T22:17:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=10862' 2024-08-31T22:17:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10862 2024-08-31T22:17:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 10862 2024-08-31T22:17:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=10960], [object_number=43, tag_tid=5, entry_tid=10959], [object_number=42, tag_tid=5, entry_tid=10958], [object_number=41, tag_tid=5, entry_tid=10957]]], state=connected] 2024-08-31T22:17:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=10960], [object_number=43, tag_tid=5, entry_tid=10959], [object_number=42, tag_tid=5, entry_tid=10958], [object_number=41, tag_tid=5, entry_tid=10957]]], state=connected] 2024-08-31T22:17:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=10960' 2024-08-31T22:17:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=10960' 2024-08-31T22:17:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=10960' '!=' 'object_number=42, tag_tid=5, entry_tid=10862' 2024-08-31T22:17:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=10960' = 'object_number=42, tag_tid=5, entry_tid=10862' 2024-08-31T22:17:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=10960' 2024-08-31T22:17:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=10960' 2024-08-31T22:17:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=10960' 2024-08-31T22:17:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10960 2024-08-31T22:17:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 10960 2024-08-31T22:17:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:44.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:44.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:44.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:44.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:44.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:44.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:44.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:44.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:44.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=11065], [object_number=40, tag_tid=5, entry_tid=11064], [object_number=43, tag_tid=5, entry_tid=11063], [object_number=42, tag_tid=5, entry_tid=11062]]], state=connected] 2024-08-31T22:17:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:44.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=11065], [object_number=40, tag_tid=5, entry_tid=11064], [object_number=43, tag_tid=5, entry_tid=11063], [object_number=42, tag_tid=5, entry_tid=11062]]], state=connected] 2024-08-31T22:17:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=11065' 2024-08-31T22:17:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=11065' 2024-08-31T22:17:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=11065' '!=' 'object_number=40, tag_tid=5, entry_tid=10960' 2024-08-31T22:17:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=11065' = 'object_number=40, tag_tid=5, entry_tid=10960' 2024-08-31T22:17:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=11065' 2024-08-31T22:17:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=11065' 2024-08-31T22:17:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=11065' 2024-08-31T22:17:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11065 2024-08-31T22:17:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 11065 2024-08-31T22:17:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=11169], [object_number=40, tag_tid=5, entry_tid=11168], [object_number=43, tag_tid=5, entry_tid=11167], [object_number=42, tag_tid=5, entry_tid=11166]]], state=connected] 2024-08-31T22:17:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:45.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:45.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:45.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:45.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:45.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=11169], [object_number=40, tag_tid=5, entry_tid=11168], [object_number=43, tag_tid=5, entry_tid=11167], [object_number=42, tag_tid=5, entry_tid=11166]]], state=connected] 2024-08-31T22:17:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=11169' 2024-08-31T22:17:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=11169' 2024-08-31T22:17:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=11169' '!=' 'object_number=41, tag_tid=5, entry_tid=11065' 2024-08-31T22:17:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=11169' = 'object_number=41, tag_tid=5, entry_tid=11065' 2024-08-31T22:17:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=11169' 2024-08-31T22:17:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=11169' 2024-08-31T22:17:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=11169' 2024-08-31T22:17:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11169 2024-08-31T22:17:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 11169 2024-08-31T22:17:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:45.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=11275], [object_number=42, tag_tid=5, entry_tid=11274], [object_number=41, tag_tid=5, entry_tid=11273], [object_number=40, tag_tid=5, entry_tid=11272]]], state=connected] 2024-08-31T22:17:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:45.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=11275], [object_number=42, tag_tid=5, entry_tid=11274], [object_number=41, tag_tid=5, entry_tid=11273], [object_number=40, tag_tid=5, entry_tid=11272]]], state=connected] 2024-08-31T22:17:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=43, tag_tid=5, entry_tid=11275' 2024-08-31T22:17:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=43, tag_tid=5, entry_tid=11275' 2024-08-31T22:17:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=11275' '!=' 'object_number=41, tag_tid=5, entry_tid=11169' 2024-08-31T22:17:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=11275' = 'object_number=41, tag_tid=5, entry_tid=11169' 2024-08-31T22:17:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=43, tag_tid=5, entry_tid=11275' 2024-08-31T22:17:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:45.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:45.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:45.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=11275' 2024-08-31T22:17:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=11275' 2024-08-31T22:17:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11275 2024-08-31T22:17:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 11275 2024-08-31T22:17:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=11378], [object_number=41, tag_tid=5, entry_tid=11377], [object_number=40, tag_tid=5, entry_tid=11376], [object_number=43, tag_tid=5, entry_tid=11375]]], state=connected] 2024-08-31T22:17:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=11378], [object_number=41, tag_tid=5, entry_tid=11377], [object_number=40, tag_tid=5, entry_tid=11376], [object_number=43, tag_tid=5, entry_tid=11375]]], state=connected] 2024-08-31T22:17:46.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=42, tag_tid=5, entry_tid=11378' 2024-08-31T22:17:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=42, tag_tid=5, entry_tid=11378' 2024-08-31T22:17:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=11378' '!=' 'object_number=43, tag_tid=5, entry_tid=11275' 2024-08-31T22:17:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=11378' = 'object_number=43, tag_tid=5, entry_tid=11275' 2024-08-31T22:17:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=42, tag_tid=5, entry_tid=11378' 2024-08-31T22:17:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=11378' 2024-08-31T22:17:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=11378' 2024-08-31T22:17:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11378 2024-08-31T22:17:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 11378 2024-08-31T22:17:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:46.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:46.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:46.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=11472], [object_number=43, tag_tid=5, entry_tid=11471], [object_number=42, tag_tid=5, entry_tid=11470], [object_number=41, tag_tid=5, entry_tid=11469]]], state=connected] 2024-08-31T22:17:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=11472], [object_number=43, tag_tid=5, entry_tid=11471], [object_number=42, tag_tid=5, entry_tid=11470], [object_number=41, tag_tid=5, entry_tid=11469]]], state=connected] 2024-08-31T22:17:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=11472' 2024-08-31T22:17:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=11472' 2024-08-31T22:17:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=11472' '!=' 'object_number=42, tag_tid=5, entry_tid=11378' 2024-08-31T22:17:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=11472' = 'object_number=42, tag_tid=5, entry_tid=11378' 2024-08-31T22:17:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=11472' 2024-08-31T22:17:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=11472' 2024-08-31T22:17:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=11472' 2024-08-31T22:17:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11472 2024-08-31T22:17:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 11472 2024-08-31T22:17:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=11561], [object_number=40, tag_tid=5, entry_tid=11560], [object_number=43, tag_tid=5, entry_tid=11559], [object_number=42, tag_tid=5, entry_tid=11558]]], state=connected] 2024-08-31T22:17:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=11561], [object_number=40, tag_tid=5, entry_tid=11560], [object_number=43, tag_tid=5, entry_tid=11559], [object_number=42, tag_tid=5, entry_tid=11558]]], state=connected] 2024-08-31T22:17:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=11561' 2024-08-31T22:17:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=11561' 2024-08-31T22:17:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=11561' '!=' 'object_number=40, tag_tid=5, entry_tid=11472' 2024-08-31T22:17:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=11561' = 'object_number=40, tag_tid=5, entry_tid=11472' 2024-08-31T22:17:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=11561' 2024-08-31T22:17:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=11561' 2024-08-31T22:17:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=11561' 2024-08-31T22:17:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:47.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11561 2024-08-31T22:17:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 11561 2024-08-31T22:17:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=11657], [object_number=40, tag_tid=5, entry_tid=11656], [object_number=43, tag_tid=5, entry_tid=11655], [object_number=42, tag_tid=5, entry_tid=11654]]], state=connected] 2024-08-31T22:17:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=11657], [object_number=40, tag_tid=5, entry_tid=11656], [object_number=43, tag_tid=5, entry_tid=11655], [object_number=42, tag_tid=5, entry_tid=11654]]], state=connected] 2024-08-31T22:17:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=11657' 2024-08-31T22:17:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=11657' 2024-08-31T22:17:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=11657' '!=' 'object_number=41, tag_tid=5, entry_tid=11561' 2024-08-31T22:17:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=11657' = 'object_number=41, tag_tid=5, entry_tid=11561' 2024-08-31T22:17:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=11657' 2024-08-31T22:17:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=11657' 2024-08-31T22:17:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=11657' 2024-08-31T22:17:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11657 2024-08-31T22:17:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 11657 2024-08-31T22:17:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:48.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:48.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:48.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=11772], [object_number=43, tag_tid=5, entry_tid=11771], [object_number=42, tag_tid=5, entry_tid=11770], [object_number=41, tag_tid=5, entry_tid=11769]]], state=connected] 2024-08-31T22:17:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=11772], [object_number=43, tag_tid=5, entry_tid=11771], [object_number=42, tag_tid=5, entry_tid=11770], [object_number=41, tag_tid=5, entry_tid=11769]]], state=connected] 2024-08-31T22:17:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=11772' 2024-08-31T22:17:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=11772' 2024-08-31T22:17:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=11772' '!=' 'object_number=41, tag_tid=5, entry_tid=11657' 2024-08-31T22:17:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=11772' = 'object_number=41, tag_tid=5, entry_tid=11657' 2024-08-31T22:17:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=11772' 2024-08-31T22:17:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=11772' 2024-08-31T22:17:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=11772' 2024-08-31T22:17:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11772 2024-08-31T22:17:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 11772 2024-08-31T22:17:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=11881], [object_number=40, tag_tid=5, entry_tid=11880], [object_number=43, tag_tid=5, entry_tid=11879], [object_number=42, tag_tid=5, entry_tid=11878]]], state=connected] 2024-08-31T22:17:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:48.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:48.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=11881], [object_number=40, tag_tid=5, entry_tid=11880], [object_number=43, tag_tid=5, entry_tid=11879], [object_number=42, tag_tid=5, entry_tid=11878]]], state=connected] 2024-08-31T22:17:48.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=11881' 2024-08-31T22:17:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=11881' 2024-08-31T22:17:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=11881' '!=' 'object_number=40, tag_tid=5, entry_tid=11772' 2024-08-31T22:17:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=11881' = 'object_number=40, tag_tid=5, entry_tid=11772' 2024-08-31T22:17:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=11881' 2024-08-31T22:17:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=11881' 2024-08-31T22:17:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=11881' 2024-08-31T22:17:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11881 2024-08-31T22:17:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 11881 2024-08-31T22:17:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=11988], [object_number=43, tag_tid=5, entry_tid=11987], [object_number=42, tag_tid=5, entry_tid=11986], [object_number=41, tag_tid=5, entry_tid=11985]]], state=connected] 2024-08-31T22:17:49.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=11988], [object_number=43, tag_tid=5, entry_tid=11987], [object_number=42, tag_tid=5, entry_tid=11986], [object_number=41, tag_tid=5, entry_tid=11985]]], state=connected] 2024-08-31T22:17:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=11988' 2024-08-31T22:17:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=11988' 2024-08-31T22:17:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=11988' '!=' 'object_number=41, tag_tid=5, entry_tid=11881' 2024-08-31T22:17:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=11988' = 'object_number=41, tag_tid=5, entry_tid=11881' 2024-08-31T22:17:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=11988' 2024-08-31T22:17:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=11988' 2024-08-31T22:17:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:49.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:49.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:49.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:49.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=11988' 2024-08-31T22:17:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11988 2024-08-31T22:17:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 11988 2024-08-31T22:17:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=12087], [object_number=42, tag_tid=5, entry_tid=12086], [object_number=41, tag_tid=5, entry_tid=12085], [object_number=40, tag_tid=5, entry_tid=12084]]], state=connected] 2024-08-31T22:17:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=12087], [object_number=42, tag_tid=5, entry_tid=12086], [object_number=41, tag_tid=5, entry_tid=12085], [object_number=40, tag_tid=5, entry_tid=12084]]], state=connected] 2024-08-31T22:17:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=43, tag_tid=5, entry_tid=12087' 2024-08-31T22:17:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=43, tag_tid=5, entry_tid=12087' 2024-08-31T22:17:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=12087' '!=' 'object_number=40, tag_tid=5, entry_tid=11988' 2024-08-31T22:17:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=12087' = 'object_number=40, tag_tid=5, entry_tid=11988' 2024-08-31T22:17:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=43, tag_tid=5, entry_tid=12087' 2024-08-31T22:17:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:49.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=12087' 2024-08-31T22:17:49.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:49.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=12087' 2024-08-31T22:17:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12087 2024-08-31T22:17:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 12087 2024-08-31T22:17:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=12198], [object_number=41, tag_tid=5, entry_tid=12197], [object_number=40, tag_tid=5, entry_tid=12196], [object_number=43, tag_tid=5, entry_tid=12195]]], state=connected] 2024-08-31T22:17:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:50.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:50.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:50.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:50.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:50.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:50.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=12198], [object_number=41, tag_tid=5, entry_tid=12197], [object_number=40, tag_tid=5, entry_tid=12196], [object_number=43, tag_tid=5, entry_tid=12195]]], state=connected] 2024-08-31T22:17:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=42, tag_tid=5, entry_tid=12198' 2024-08-31T22:17:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=42, tag_tid=5, entry_tid=12198' 2024-08-31T22:17:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=12198' '!=' 'object_number=43, tag_tid=5, entry_tid=12087' 2024-08-31T22:17:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=12198' = 'object_number=43, tag_tid=5, entry_tid=12087' 2024-08-31T22:17:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=42, tag_tid=5, entry_tid=12198' 2024-08-31T22:17:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=12198' 2024-08-31T22:17:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=12198' 2024-08-31T22:17:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12198 2024-08-31T22:17:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 12198 2024-08-31T22:17:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=12312], [object_number=43, tag_tid=5, entry_tid=12311], [object_number=42, tag_tid=5, entry_tid=12310], [object_number=41, tag_tid=5, entry_tid=12309]]], state=connected] 2024-08-31T22:17:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=12312], [object_number=43, tag_tid=5, entry_tid=12311], [object_number=42, tag_tid=5, entry_tid=12310], [object_number=41, tag_tid=5, entry_tid=12309]]], state=connected] 2024-08-31T22:17:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=12312' 2024-08-31T22:17:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=12312' 2024-08-31T22:17:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=12312' '!=' 'object_number=42, tag_tid=5, entry_tid=12198' 2024-08-31T22:17:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=12312' = 'object_number=42, tag_tid=5, entry_tid=12198' 2024-08-31T22:17:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=12312' 2024-08-31T22:17:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:50.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:50.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=12312' 2024-08-31T22:17:50.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:50.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=12312' 2024-08-31T22:17:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12312 2024-08-31T22:17:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 12312 2024-08-31T22:17:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=12423], [object_number=42, tag_tid=5, entry_tid=12422], [object_number=41, tag_tid=5, entry_tid=12421], [object_number=40, tag_tid=5, entry_tid=12420]]], state=connected] 2024-08-31T22:17:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=12423], [object_number=42, tag_tid=5, entry_tid=12422], [object_number=41, tag_tid=5, entry_tid=12421], [object_number=40, tag_tid=5, entry_tid=12420]]], state=connected] 2024-08-31T22:17:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=43, tag_tid=5, entry_tid=12423' 2024-08-31T22:17:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=43, tag_tid=5, entry_tid=12423' 2024-08-31T22:17:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=12423' '!=' 'object_number=40, tag_tid=5, entry_tid=12312' 2024-08-31T22:17:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=12423' = 'object_number=40, tag_tid=5, entry_tid=12312' 2024-08-31T22:17:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=43, tag_tid=5, entry_tid=12423' 2024-08-31T22:17:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=12423' 2024-08-31T22:17:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=12423' 2024-08-31T22:17:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12423 2024-08-31T22:17:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 12423 2024-08-31T22:17:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:51.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:51.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:51.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=12530], [object_number=41, tag_tid=5, entry_tid=12529], [object_number=40, tag_tid=5, entry_tid=12528], [object_number=43, tag_tid=5, entry_tid=12527]]], state=connected] 2024-08-31T22:17:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:51.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:51.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=12530], [object_number=41, tag_tid=5, entry_tid=12529], [object_number=40, tag_tid=5, entry_tid=12528], [object_number=43, tag_tid=5, entry_tid=12527]]], state=connected] 2024-08-31T22:17:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=42, tag_tid=5, entry_tid=12530' 2024-08-31T22:17:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=42, tag_tid=5, entry_tid=12530' 2024-08-31T22:17:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=12530' '!=' 'object_number=43, tag_tid=5, entry_tid=12423' 2024-08-31T22:17:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=12530' = 'object_number=43, tag_tid=5, entry_tid=12423' 2024-08-31T22:17:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=42, tag_tid=5, entry_tid=12530' 2024-08-31T22:17:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=12530' 2024-08-31T22:17:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=12530' 2024-08-31T22:17:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12530 2024-08-31T22:17:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 12530 2024-08-31T22:17:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:51.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:51.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:51.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=12638], [object_number=41, tag_tid=5, entry_tid=12637], [object_number=40, tag_tid=5, entry_tid=12636], [object_number=43, tag_tid=5, entry_tid=12635]]], state=connected] 2024-08-31T22:17:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=12638], [object_number=41, tag_tid=5, entry_tid=12637], [object_number=40, tag_tid=5, entry_tid=12636], [object_number=43, tag_tid=5, entry_tid=12635]]], state=connected] 2024-08-31T22:17:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=42, tag_tid=5, entry_tid=12638' 2024-08-31T22:17:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=42, tag_tid=5, entry_tid=12638' 2024-08-31T22:17:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=12638' '!=' 'object_number=42, tag_tid=5, entry_tid=12530' 2024-08-31T22:17:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=12638' = 'object_number=42, tag_tid=5, entry_tid=12530' 2024-08-31T22:17:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=42, tag_tid=5, entry_tid=12638' 2024-08-31T22:17:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=12638' 2024-08-31T22:17:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=12638' 2024-08-31T22:17:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12638 2024-08-31T22:17:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 12638 2024-08-31T22:17:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:52.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:52.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:52.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:52.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:52.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:52.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:52.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:52.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:52.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:52.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:52.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:52.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:52.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:52.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:52.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=12744], [object_number=43, tag_tid=5, entry_tid=12743], [object_number=42, tag_tid=5, entry_tid=12742], [object_number=41, tag_tid=5, entry_tid=12741]]], state=connected] 2024-08-31T22:17:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=12744], [object_number=43, tag_tid=5, entry_tid=12743], [object_number=42, tag_tid=5, entry_tid=12742], [object_number=41, tag_tid=5, entry_tid=12741]]], state=connected] 2024-08-31T22:17:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=12744' 2024-08-31T22:17:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=12744' 2024-08-31T22:17:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=12744' '!=' 'object_number=42, tag_tid=5, entry_tid=12638' 2024-08-31T22:17:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=12744' = 'object_number=42, tag_tid=5, entry_tid=12638' 2024-08-31T22:17:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=12744' 2024-08-31T22:17:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=12744' 2024-08-31T22:17:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=12744' 2024-08-31T22:17:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12744 2024-08-31T22:17:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 12744 2024-08-31T22:17:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:52.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:52.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=12864], [object_number=43, tag_tid=5, entry_tid=12863], [object_number=42, tag_tid=5, entry_tid=12862], [object_number=41, tag_tid=5, entry_tid=12861]]], state=connected] 2024-08-31T22:17:52.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:52.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:52.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:52.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:52.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:52.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=12864], [object_number=43, tag_tid=5, entry_tid=12863], [object_number=42, tag_tid=5, entry_tid=12862], [object_number=41, tag_tid=5, entry_tid=12861]]], state=connected] 2024-08-31T22:17:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=12864' 2024-08-31T22:17:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=12864' 2024-08-31T22:17:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=12864' '!=' 'object_number=40, tag_tid=5, entry_tid=12744' 2024-08-31T22:17:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=12864' = 'object_number=40, tag_tid=5, entry_tid=12744' 2024-08-31T22:17:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=12864' 2024-08-31T22:17:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=12864' 2024-08-31T22:17:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=12864' 2024-08-31T22:17:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12864 2024-08-31T22:17:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 12864 2024-08-31T22:17:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=12978], [object_number=41, tag_tid=5, entry_tid=12977], [object_number=40, tag_tid=5, entry_tid=12976], [object_number=43, tag_tid=5, entry_tid=12975]]], state=connected] 2024-08-31T22:17:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=12978], [object_number=41, tag_tid=5, entry_tid=12977], [object_number=40, tag_tid=5, entry_tid=12976], [object_number=43, tag_tid=5, entry_tid=12975]]], state=connected] 2024-08-31T22:17:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=42, tag_tid=5, entry_tid=12978' 2024-08-31T22:17:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=42, tag_tid=5, entry_tid=12978' 2024-08-31T22:17:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=12978' '!=' 'object_number=40, tag_tid=5, entry_tid=12864' 2024-08-31T22:17:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=12978' = 'object_number=40, tag_tid=5, entry_tid=12864' 2024-08-31T22:17:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=42, tag_tid=5, entry_tid=12978' 2024-08-31T22:17:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:53.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=12978' 2024-08-31T22:17:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=12978' 2024-08-31T22:17:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12978 2024-08-31T22:17:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 12978 2024-08-31T22:17:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:53.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:53.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:53.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:53.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=13081], [object_number=40, tag_tid=5, entry_tid=13080], [object_number=43, tag_tid=5, entry_tid=13079], [object_number=42, tag_tid=5, entry_tid=13078]]], state=connected] 2024-08-31T22:17:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=13081], [object_number=40, tag_tid=5, entry_tid=13080], [object_number=43, tag_tid=5, entry_tid=13079], [object_number=42, tag_tid=5, entry_tid=13078]]], state=connected] 2024-08-31T22:17:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=13081' 2024-08-31T22:17:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=13081' 2024-08-31T22:17:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=13081' '!=' 'object_number=42, tag_tid=5, entry_tid=12978' 2024-08-31T22:17:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=13081' = 'object_number=42, tag_tid=5, entry_tid=12978' 2024-08-31T22:17:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=13081' 2024-08-31T22:17:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=13081' 2024-08-31T22:17:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=13081' 2024-08-31T22:17:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13081 2024-08-31T22:17:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 13081 2024-08-31T22:17:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=13189], [object_number=40, tag_tid=5, entry_tid=13188], [object_number=43, tag_tid=5, entry_tid=13187], [object_number=42, tag_tid=5, entry_tid=13186]]], state=connected] 2024-08-31T22:17:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=13189], [object_number=40, tag_tid=5, entry_tid=13188], [object_number=43, tag_tid=5, entry_tid=13187], [object_number=42, tag_tid=5, entry_tid=13186]]], state=connected] 2024-08-31T22:17:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=13189' 2024-08-31T22:17:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=13189' 2024-08-31T22:17:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=13189' '!=' 'object_number=41, tag_tid=5, entry_tid=13081' 2024-08-31T22:17:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=13189' = 'object_number=41, tag_tid=5, entry_tid=13081' 2024-08-31T22:17:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=13189' 2024-08-31T22:17:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=13189' 2024-08-31T22:17:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=13189' 2024-08-31T22:17:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13189 2024-08-31T22:17:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 13189 2024-08-31T22:17:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:54.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:54.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=13301], [object_number=40, tag_tid=5, entry_tid=13300], [object_number=43, tag_tid=5, entry_tid=13299], [object_number=42, tag_tid=5, entry_tid=13298]]], state=connected] 2024-08-31T22:17:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=13301], [object_number=40, tag_tid=5, entry_tid=13300], [object_number=43, tag_tid=5, entry_tid=13299], [object_number=42, tag_tid=5, entry_tid=13298]]], state=connected] 2024-08-31T22:17:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=13301' 2024-08-31T22:17:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=13301' 2024-08-31T22:17:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=13301' '!=' 'object_number=41, tag_tid=5, entry_tid=13189' 2024-08-31T22:17:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=13301' = 'object_number=41, tag_tid=5, entry_tid=13189' 2024-08-31T22:17:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=13301' 2024-08-31T22:17:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=13301' 2024-08-31T22:17:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=13301' 2024-08-31T22:17:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13301 2024-08-31T22:17:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 13301 2024-08-31T22:17:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:55.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=13436], [object_number=43, tag_tid=5, entry_tid=13435], [object_number=42, tag_tid=5, entry_tid=13434], [object_number=41, tag_tid=5, entry_tid=13433]]], state=connected] 2024-08-31T22:17:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=13436], [object_number=43, tag_tid=5, entry_tid=13435], [object_number=42, tag_tid=5, entry_tid=13434], [object_number=41, tag_tid=5, entry_tid=13433]]], state=connected] 2024-08-31T22:17:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=13436' 2024-08-31T22:17:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=13436' 2024-08-31T22:17:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=13436' '!=' 'object_number=41, tag_tid=5, entry_tid=13301' 2024-08-31T22:17:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=13436' = 'object_number=41, tag_tid=5, entry_tid=13301' 2024-08-31T22:17:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=13436' 2024-08-31T22:17:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=13436' 2024-08-31T22:17:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:55.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=13436' 2024-08-31T22:17:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13436 2024-08-31T22:17:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 13436 2024-08-31T22:17:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:55.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=13620], [object_number=43, tag_tid=5, entry_tid=13619], [object_number=42, tag_tid=5, entry_tid=13618], [object_number=41, tag_tid=5, entry_tid=13617]]], state=connected] 2024-08-31T22:17:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:55.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=13620], [object_number=43, tag_tid=5, entry_tid=13619], [object_number=42, tag_tid=5, entry_tid=13618], [object_number=41, tag_tid=5, entry_tid=13617]]], state=connected] 2024-08-31T22:17:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=13620' 2024-08-31T22:17:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=13620' 2024-08-31T22:17:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=13620' '!=' 'object_number=40, tag_tid=5, entry_tid=13436' 2024-08-31T22:17:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=13620' = 'object_number=40, tag_tid=5, entry_tid=13436' 2024-08-31T22:17:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=13620' 2024-08-31T22:17:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=13620' 2024-08-31T22:17:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=13620' 2024-08-31T22:17:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13620 2024-08-31T22:17:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 13620 2024-08-31T22:17:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:56.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:56.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:56.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:56.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:56.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=13750], [object_number=41, tag_tid=5, entry_tid=13749], [object_number=40, tag_tid=5, entry_tid=13748], [object_number=43, tag_tid=5, entry_tid=13747]]], state=connected] 2024-08-31T22:17:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:56.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:56.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:56.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:56.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=13750], [object_number=41, tag_tid=5, entry_tid=13749], [object_number=40, tag_tid=5, entry_tid=13748], [object_number=43, tag_tid=5, entry_tid=13747]]], state=connected] 2024-08-31T22:17:56.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=42, tag_tid=5, entry_tid=13750' 2024-08-31T22:17:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=42, tag_tid=5, entry_tid=13750' 2024-08-31T22:17:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=13750' '!=' 'object_number=40, tag_tid=5, entry_tid=13620' 2024-08-31T22:17:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=13750' = 'object_number=40, tag_tid=5, entry_tid=13620' 2024-08-31T22:17:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=42, tag_tid=5, entry_tid=13750' 2024-08-31T22:17:56.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:56.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:56.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:56.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=13750' 2024-08-31T22:17:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=13750' 2024-08-31T22:17:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13750 2024-08-31T22:17:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 13750 2024-08-31T22:17:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:56.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:56.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:56.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:56.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=13873], [object_number=40, tag_tid=5, entry_tid=13872], [object_number=43, tag_tid=5, entry_tid=13871], [object_number=42, tag_tid=5, entry_tid=13870]]], state=connected] 2024-08-31T22:17:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:56.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:56.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:56.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:56.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:56.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=13873], [object_number=40, tag_tid=5, entry_tid=13872], [object_number=43, tag_tid=5, entry_tid=13871], [object_number=42, tag_tid=5, entry_tid=13870]]], state=connected] 2024-08-31T22:17:56.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=13873' 2024-08-31T22:17:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=13873' 2024-08-31T22:17:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=13873' '!=' 'object_number=42, tag_tid=5, entry_tid=13750' 2024-08-31T22:17:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=13873' = 'object_number=42, tag_tid=5, entry_tid=13750' 2024-08-31T22:17:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=13873' 2024-08-31T22:17:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=13873' 2024-08-31T22:17:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=13873' 2024-08-31T22:17:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13873 2024-08-31T22:17:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 13873 2024-08-31T22:17:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=13940], [object_number=43, tag_tid=5, entry_tid=13939], [object_number=42, tag_tid=5, entry_tid=13938], [object_number=41, tag_tid=5, entry_tid=13937]]], state=connected] 2024-08-31T22:17:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 10 2024-08-31T22:17:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=13940], [object_number=43, tag_tid=5, entry_tid=13939], [object_number=42, tag_tid=5, entry_tid=13938], [object_number=41, tag_tid=5, entry_tid=13937]]], state=connected] 2024-08-31T22:17:57.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=13940' 2024-08-31T22:17:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=13940' 2024-08-31T22:17:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=13940' '!=' 'object_number=41, tag_tid=5, entry_tid=13873' 2024-08-31T22:17:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=13940' = 'object_number=41, tag_tid=5, entry_tid=13873' 2024-08-31T22:17:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=13940' 2024-08-31T22:17:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=13940' 2024-08-31T22:17:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=13940' 2024-08-31T22:17:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13940 2024-08-31T22:17:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 13940 2024-08-31T22:17:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:17:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=13987], [object_number=46, tag_tid=5, entry_tid=13986], [object_number=45, tag_tid=5, entry_tid=13985], [object_number=44, tag_tid=5, entry_tid=13984]]], state=connected] 2024-08-31T22:17:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:17:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=13987], [object_number=46, tag_tid=5, entry_tid=13986], [object_number=45, tag_tid=5, entry_tid=13985], [object_number=44, tag_tid=5, entry_tid=13984]]], state=connected] 2024-08-31T22:17:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=47, tag_tid=5, entry_tid=13987' 2024-08-31T22:17:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=47, tag_tid=5, entry_tid=13987' 2024-08-31T22:17:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=13987' '!=' 'object_number=44, tag_tid=5, entry_tid=13940' 2024-08-31T22:17:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=13987' = 'object_number=44, tag_tid=5, entry_tid=13940' 2024-08-31T22:17:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=47, tag_tid=5, entry_tid=13987' 2024-08-31T22:17:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=13987' 2024-08-31T22:17:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=13987' 2024-08-31T22:17:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13987 2024-08-31T22:17:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 13987 2024-08-31T22:17:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:17:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=14086], [object_number=45, tag_tid=5, entry_tid=14085], [object_number=44, tag_tid=5, entry_tid=14084], [object_number=47, tag_tid=5, entry_tid=14083]]], state=connected] 2024-08-31T22:17:58.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:17:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=14086], [object_number=45, tag_tid=5, entry_tid=14085], [object_number=44, tag_tid=5, entry_tid=14084], [object_number=47, tag_tid=5, entry_tid=14083]]], state=connected] 2024-08-31T22:17:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=14086' 2024-08-31T22:17:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=14086' 2024-08-31T22:17:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=14086' '!=' 'object_number=47, tag_tid=5, entry_tid=13987' 2024-08-31T22:17:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=14086' = 'object_number=47, tag_tid=5, entry_tid=13987' 2024-08-31T22:17:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=14086' 2024-08-31T22:17:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=14086' 2024-08-31T22:17:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=14086' 2024-08-31T22:17:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14086 2024-08-31T22:17:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 14086 2024-08-31T22:17:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:17:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=14183], [object_number=46, tag_tid=5, entry_tid=14182], [object_number=45, tag_tid=5, entry_tid=14181], [object_number=44, tag_tid=5, entry_tid=14180]]], state=connected] 2024-08-31T22:17:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:17:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=14183], [object_number=46, tag_tid=5, entry_tid=14182], [object_number=45, tag_tid=5, entry_tid=14181], [object_number=44, tag_tid=5, entry_tid=14180]]], state=connected] 2024-08-31T22:17:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=47, tag_tid=5, entry_tid=14183' 2024-08-31T22:17:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=47, tag_tid=5, entry_tid=14183' 2024-08-31T22:17:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=14183' '!=' 'object_number=46, tag_tid=5, entry_tid=14086' 2024-08-31T22:17:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=14183' = 'object_number=46, tag_tid=5, entry_tid=14086' 2024-08-31T22:17:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=47, tag_tid=5, entry_tid=14183' 2024-08-31T22:17:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=14183' 2024-08-31T22:17:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=14183' 2024-08-31T22:17:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14183 2024-08-31T22:17:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 14183 2024-08-31T22:17:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:17:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=14280], [object_number=47, tag_tid=5, entry_tid=14279], [object_number=46, tag_tid=5, entry_tid=14278], [object_number=45, tag_tid=5, entry_tid=14277]]], state=connected] 2024-08-31T22:17:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:17:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=14280], [object_number=47, tag_tid=5, entry_tid=14279], [object_number=46, tag_tid=5, entry_tid=14278], [object_number=45, tag_tid=5, entry_tid=14277]]], state=connected] 2024-08-31T22:17:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=14280' 2024-08-31T22:17:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=14280' 2024-08-31T22:17:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=14280' '!=' 'object_number=47, tag_tid=5, entry_tid=14183' 2024-08-31T22:17:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=14280' = 'object_number=47, tag_tid=5, entry_tid=14183' 2024-08-31T22:17:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=14280' 2024-08-31T22:17:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=14280' 2024-08-31T22:17:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=14280' 2024-08-31T22:17:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14280 2024-08-31T22:17:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 14280 2024-08-31T22:17:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:17:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:17:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:17:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:17:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:17:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:17:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:17:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:17:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:17:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:17:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:17:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:17:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:17:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:17:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:17:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:17:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:17:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:17:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:17:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:17:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:17:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:17:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:17:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:17:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:17:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:17:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:17:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:17:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:17:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:17:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:17:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:17:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:17:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:17:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:17:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:17:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:17:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:17:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:17:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:17:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:17:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:17:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:17:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:17:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:17:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:17:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:17:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:17:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:17:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:17:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:17:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:17:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:59.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:59.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=14375], [object_number=46, tag_tid=5, entry_tid=14374], [object_number=45, tag_tid=5, entry_tid=14373], [object_number=44, tag_tid=5, entry_tid=14372]]], state=connected] 2024-08-31T22:17:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:17:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:17:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:17:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:17:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:17:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:17:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:17:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:17:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:17:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:17:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:59.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:17:59.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:17:59.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:17:59.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:17:59.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=14375], [object_number=46, tag_tid=5, entry_tid=14374], [object_number=45, tag_tid=5, entry_tid=14373], [object_number=44, tag_tid=5, entry_tid=14372]]], state=connected] 2024-08-31T22:17:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:17:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=47, tag_tid=5, entry_tid=14375' 2024-08-31T22:17:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=47, tag_tid=5, entry_tid=14375' 2024-08-31T22:17:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=14375' '!=' 'object_number=44, tag_tid=5, entry_tid=14280' 2024-08-31T22:17:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:17:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=14375' = 'object_number=44, tag_tid=5, entry_tid=14280' 2024-08-31T22:17:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=47, tag_tid=5, entry_tid=14375' 2024-08-31T22:17:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:17:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=14375' 2024-08-31T22:17:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:17:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:17:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:17:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:17:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=14375' 2024-08-31T22:17:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:17:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:17:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14375 2024-08-31T22:17:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 14375 2024-08-31T22:17:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:17:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:17:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:00.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:00.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=14480], [object_number=47, tag_tid=5, entry_tid=14479], [object_number=46, tag_tid=5, entry_tid=14478], [object_number=45, tag_tid=5, entry_tid=14477]]], state=connected] 2024-08-31T22:18:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=14480], [object_number=47, tag_tid=5, entry_tid=14479], [object_number=46, tag_tid=5, entry_tid=14478], [object_number=45, tag_tid=5, entry_tid=14477]]], state=connected] 2024-08-31T22:18:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=14480' 2024-08-31T22:18:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=14480' 2024-08-31T22:18:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=14480' '!=' 'object_number=47, tag_tid=5, entry_tid=14375' 2024-08-31T22:18:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=14480' = 'object_number=47, tag_tid=5, entry_tid=14375' 2024-08-31T22:18:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=14480' 2024-08-31T22:18:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=14480' 2024-08-31T22:18:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=14480' 2024-08-31T22:18:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14480 2024-08-31T22:18:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 14480 2024-08-31T22:18:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:00.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:00.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:00.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:00.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:00.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=14585], [object_number=44, tag_tid=5, entry_tid=14584], [object_number=47, tag_tid=5, entry_tid=14583], [object_number=46, tag_tid=5, entry_tid=14582]]], state=connected] 2024-08-31T22:18:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=14585], [object_number=44, tag_tid=5, entry_tid=14584], [object_number=47, tag_tid=5, entry_tid=14583], [object_number=46, tag_tid=5, entry_tid=14582]]], state=connected] 2024-08-31T22:18:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=45, tag_tid=5, entry_tid=14585' 2024-08-31T22:18:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=45, tag_tid=5, entry_tid=14585' 2024-08-31T22:18:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=14585' '!=' 'object_number=44, tag_tid=5, entry_tid=14480' 2024-08-31T22:18:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=14585' = 'object_number=44, tag_tid=5, entry_tid=14480' 2024-08-31T22:18:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=45, tag_tid=5, entry_tid=14585' 2024-08-31T22:18:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=14585' 2024-08-31T22:18:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=14585' 2024-08-31T22:18:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14585 2024-08-31T22:18:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 14585 2024-08-31T22:18:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:01.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=14674], [object_number=45, tag_tid=5, entry_tid=14673], [object_number=44, tag_tid=5, entry_tid=14672], [object_number=47, tag_tid=5, entry_tid=14671]]], state=connected] 2024-08-31T22:18:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:01.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:01.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:01.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:01.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:01.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=14674], [object_number=45, tag_tid=5, entry_tid=14673], [object_number=44, tag_tid=5, entry_tid=14672], [object_number=47, tag_tid=5, entry_tid=14671]]], state=connected] 2024-08-31T22:18:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=14674' 2024-08-31T22:18:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=14674' 2024-08-31T22:18:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=14674' '!=' 'object_number=45, tag_tid=5, entry_tid=14585' 2024-08-31T22:18:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=14674' = 'object_number=45, tag_tid=5, entry_tid=14585' 2024-08-31T22:18:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=14674' 2024-08-31T22:18:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=14674' 2024-08-31T22:18:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=14674' 2024-08-31T22:18:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14674 2024-08-31T22:18:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 14674 2024-08-31T22:18:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=14768], [object_number=47, tag_tid=5, entry_tid=14767], [object_number=46, tag_tid=5, entry_tid=14766], [object_number=45, tag_tid=5, entry_tid=14765]]], state=connected] 2024-08-31T22:18:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=14768], [object_number=47, tag_tid=5, entry_tid=14767], [object_number=46, tag_tid=5, entry_tid=14766], [object_number=45, tag_tid=5, entry_tid=14765]]], state=connected] 2024-08-31T22:18:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=14768' 2024-08-31T22:18:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=14768' 2024-08-31T22:18:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=14768' '!=' 'object_number=46, tag_tid=5, entry_tid=14674' 2024-08-31T22:18:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=14768' = 'object_number=46, tag_tid=5, entry_tid=14674' 2024-08-31T22:18:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=14768' 2024-08-31T22:18:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=14768' 2024-08-31T22:18:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=14768' 2024-08-31T22:18:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14768 2024-08-31T22:18:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 14768 2024-08-31T22:18:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:02.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:02.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:02.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:02.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:02.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:02.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:02.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:02.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:02.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:02.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=14867], [object_number=46, tag_tid=5, entry_tid=14866], [object_number=45, tag_tid=5, entry_tid=14865], [object_number=44, tag_tid=5, entry_tid=14864]]], state=connected] 2024-08-31T22:18:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=14867], [object_number=46, tag_tid=5, entry_tid=14866], [object_number=45, tag_tid=5, entry_tid=14865], [object_number=44, tag_tid=5, entry_tid=14864]]], state=connected] 2024-08-31T22:18:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=47, tag_tid=5, entry_tid=14867' 2024-08-31T22:18:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=47, tag_tid=5, entry_tid=14867' 2024-08-31T22:18:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=14867' '!=' 'object_number=44, tag_tid=5, entry_tid=14768' 2024-08-31T22:18:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=14867' = 'object_number=44, tag_tid=5, entry_tid=14768' 2024-08-31T22:18:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=47, tag_tid=5, entry_tid=14867' 2024-08-31T22:18:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:02.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=14867' 2024-08-31T22:18:02.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:02.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:02.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:02.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=14867' 2024-08-31T22:18:02.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:02.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14867 2024-08-31T22:18:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 14867 2024-08-31T22:18:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:02.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:02.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=14960], [object_number=47, tag_tid=5, entry_tid=14959], [object_number=46, tag_tid=5, entry_tid=14958], [object_number=45, tag_tid=5, entry_tid=14957]]], state=connected] 2024-08-31T22:18:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:02.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:02.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:02.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:02.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:02.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:02.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:02.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:02.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:02.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:02.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:02.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=14960], [object_number=47, tag_tid=5, entry_tid=14959], [object_number=46, tag_tid=5, entry_tid=14958], [object_number=45, tag_tid=5, entry_tid=14957]]], state=connected] 2024-08-31T22:18:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=14960' 2024-08-31T22:18:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=14960' 2024-08-31T22:18:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=14960' '!=' 'object_number=47, tag_tid=5, entry_tid=14867' 2024-08-31T22:18:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=14960' = 'object_number=47, tag_tid=5, entry_tid=14867' 2024-08-31T22:18:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=14960' 2024-08-31T22:18:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=14960' 2024-08-31T22:18:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=14960' 2024-08-31T22:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14960 2024-08-31T22:18:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 14960 2024-08-31T22:18:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=15060], [object_number=47, tag_tid=5, entry_tid=15059], [object_number=46, tag_tid=5, entry_tid=15058], [object_number=45, tag_tid=5, entry_tid=15057]]], state=connected] 2024-08-31T22:18:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=15060], [object_number=47, tag_tid=5, entry_tid=15059], [object_number=46, tag_tid=5, entry_tid=15058], [object_number=45, tag_tid=5, entry_tid=15057]]], state=connected] 2024-08-31T22:18:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=15060' 2024-08-31T22:18:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=15060' 2024-08-31T22:18:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=15060' '!=' 'object_number=44, tag_tid=5, entry_tid=14960' 2024-08-31T22:18:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=15060' = 'object_number=44, tag_tid=5, entry_tid=14960' 2024-08-31T22:18:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=15060' 2024-08-31T22:18:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=15060' 2024-08-31T22:18:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=15060' 2024-08-31T22:18:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15060 2024-08-31T22:18:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 15060 2024-08-31T22:18:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:03.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:03.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=15157], [object_number=44, tag_tid=5, entry_tid=15156], [object_number=47, tag_tid=5, entry_tid=15155], [object_number=46, tag_tid=5, entry_tid=15154]]], state=connected] 2024-08-31T22:18:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:03.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:03.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:03.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=15157], [object_number=44, tag_tid=5, entry_tid=15156], [object_number=47, tag_tid=5, entry_tid=15155], [object_number=46, tag_tid=5, entry_tid=15154]]], state=connected] 2024-08-31T22:18:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=45, tag_tid=5, entry_tid=15157' 2024-08-31T22:18:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=45, tag_tid=5, entry_tid=15157' 2024-08-31T22:18:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=15157' '!=' 'object_number=44, tag_tid=5, entry_tid=15060' 2024-08-31T22:18:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=15157' = 'object_number=44, tag_tid=5, entry_tid=15060' 2024-08-31T22:18:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=45, tag_tid=5, entry_tid=15157' 2024-08-31T22:18:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=15157' 2024-08-31T22:18:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=15157' 2024-08-31T22:18:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15157 2024-08-31T22:18:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 15157 2024-08-31T22:18:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:03.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:03.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:03.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:03.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:03.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:03.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:03.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:03.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:03.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:03.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:03.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:03.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:04.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:04.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:04.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=15249], [object_number=44, tag_tid=5, entry_tid=15248], [object_number=47, tag_tid=5, entry_tid=15247], [object_number=46, tag_tid=5, entry_tid=15246]]], state=connected] 2024-08-31T22:18:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=15249], [object_number=44, tag_tid=5, entry_tid=15248], [object_number=47, tag_tid=5, entry_tid=15247], [object_number=46, tag_tid=5, entry_tid=15246]]], state=connected] 2024-08-31T22:18:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=45, tag_tid=5, entry_tid=15249' 2024-08-31T22:18:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=45, tag_tid=5, entry_tid=15249' 2024-08-31T22:18:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=15249' '!=' 'object_number=45, tag_tid=5, entry_tid=15157' 2024-08-31T22:18:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=15249' = 'object_number=45, tag_tid=5, entry_tid=15157' 2024-08-31T22:18:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=45, tag_tid=5, entry_tid=15249' 2024-08-31T22:18:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=15249' 2024-08-31T22:18:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:04.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=15249' 2024-08-31T22:18:04.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:04.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15249 2024-08-31T22:18:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 15249 2024-08-31T22:18:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=15342], [object_number=45, tag_tid=5, entry_tid=15341], [object_number=44, tag_tid=5, entry_tid=15340], [object_number=47, tag_tid=5, entry_tid=15339]]], state=connected] 2024-08-31T22:18:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=15342], [object_number=45, tag_tid=5, entry_tid=15341], [object_number=44, tag_tid=5, entry_tid=15340], [object_number=47, tag_tid=5, entry_tid=15339]]], state=connected] 2024-08-31T22:18:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=15342' 2024-08-31T22:18:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=15342' 2024-08-31T22:18:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=15342' '!=' 'object_number=45, tag_tid=5, entry_tid=15249' 2024-08-31T22:18:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=15342' = 'object_number=45, tag_tid=5, entry_tid=15249' 2024-08-31T22:18:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=15342' 2024-08-31T22:18:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=15342' 2024-08-31T22:18:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=15342' 2024-08-31T22:18:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15342 2024-08-31T22:18:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 15342 2024-08-31T22:18:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:04.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:05.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:05.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:05.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:05.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:05.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:05.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:05.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:05.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=15435], [object_number=46, tag_tid=5, entry_tid=15434], [object_number=45, tag_tid=5, entry_tid=15433], [object_number=44, tag_tid=5, entry_tid=15432]]], state=connected] 2024-08-31T22:18:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=15435], [object_number=46, tag_tid=5, entry_tid=15434], [object_number=45, tag_tid=5, entry_tid=15433], [object_number=44, tag_tid=5, entry_tid=15432]]], state=connected] 2024-08-31T22:18:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=47, tag_tid=5, entry_tid=15435' 2024-08-31T22:18:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=47, tag_tid=5, entry_tid=15435' 2024-08-31T22:18:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=15435' '!=' 'object_number=46, tag_tid=5, entry_tid=15342' 2024-08-31T22:18:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=15435' = 'object_number=46, tag_tid=5, entry_tid=15342' 2024-08-31T22:18:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=47, tag_tid=5, entry_tid=15435' 2024-08-31T22:18:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=15435' 2024-08-31T22:18:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=15435' 2024-08-31T22:18:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15435 2024-08-31T22:18:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 15435 2024-08-31T22:18:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=15536], [object_number=47, tag_tid=5, entry_tid=15535], [object_number=46, tag_tid=5, entry_tid=15534], [object_number=45, tag_tid=5, entry_tid=15533]]], state=connected] 2024-08-31T22:18:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=15536], [object_number=47, tag_tid=5, entry_tid=15535], [object_number=46, tag_tid=5, entry_tid=15534], [object_number=45, tag_tid=5, entry_tid=15533]]], state=connected] 2024-08-31T22:18:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=15536' 2024-08-31T22:18:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=15536' 2024-08-31T22:18:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=15536' '!=' 'object_number=47, tag_tid=5, entry_tid=15435' 2024-08-31T22:18:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=15536' = 'object_number=47, tag_tid=5, entry_tid=15435' 2024-08-31T22:18:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=15536' 2024-08-31T22:18:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=15536' 2024-08-31T22:18:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=15536' 2024-08-31T22:18:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15536 2024-08-31T22:18:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 15536 2024-08-31T22:18:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:06.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=15652], [object_number=47, tag_tid=5, entry_tid=15651], [object_number=46, tag_tid=5, entry_tid=15650], [object_number=45, tag_tid=5, entry_tid=15649]]], state=connected] 2024-08-31T22:18:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=15652], [object_number=47, tag_tid=5, entry_tid=15651], [object_number=46, tag_tid=5, entry_tid=15650], [object_number=45, tag_tid=5, entry_tid=15649]]], state=connected] 2024-08-31T22:18:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=15652' 2024-08-31T22:18:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=15652' 2024-08-31T22:18:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=15652' '!=' 'object_number=44, tag_tid=5, entry_tid=15536' 2024-08-31T22:18:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=15652' = 'object_number=44, tag_tid=5, entry_tid=15536' 2024-08-31T22:18:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=15652' 2024-08-31T22:18:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=15652' 2024-08-31T22:18:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=15652' 2024-08-31T22:18:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15652 2024-08-31T22:18:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 15652 2024-08-31T22:18:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=15764], [object_number=47, tag_tid=5, entry_tid=15763], [object_number=46, tag_tid=5, entry_tid=15762], [object_number=45, tag_tid=5, entry_tid=15761]]], state=connected] 2024-08-31T22:18:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=15764], [object_number=47, tag_tid=5, entry_tid=15763], [object_number=46, tag_tid=5, entry_tid=15762], [object_number=45, tag_tid=5, entry_tid=15761]]], state=connected] 2024-08-31T22:18:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:06.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=15764' 2024-08-31T22:18:06.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=15764' 2024-08-31T22:18:06.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=15764' '!=' 'object_number=44, tag_tid=5, entry_tid=15652' 2024-08-31T22:18:06.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=15764' = 'object_number=44, tag_tid=5, entry_tid=15652' 2024-08-31T22:18:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=15764' 2024-08-31T22:18:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=15764' 2024-08-31T22:18:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:06.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=15764' 2024-08-31T22:18:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15764 2024-08-31T22:18:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 15764 2024-08-31T22:18:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=15884], [object_number=47, tag_tid=5, entry_tid=15883], [object_number=46, tag_tid=5, entry_tid=15882], [object_number=45, tag_tid=5, entry_tid=15881]]], state=connected] 2024-08-31T22:18:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=15884], [object_number=47, tag_tid=5, entry_tid=15883], [object_number=46, tag_tid=5, entry_tid=15882], [object_number=45, tag_tid=5, entry_tid=15881]]], state=connected] 2024-08-31T22:18:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=15884' 2024-08-31T22:18:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=15884' 2024-08-31T22:18:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=15884' '!=' 'object_number=44, tag_tid=5, entry_tid=15764' 2024-08-31T22:18:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=15884' = 'object_number=44, tag_tid=5, entry_tid=15764' 2024-08-31T22:18:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=15884' 2024-08-31T22:18:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=15884' 2024-08-31T22:18:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:07.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:07.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:07.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=15884' 2024-08-31T22:18:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15884 2024-08-31T22:18:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 15884 2024-08-31T22:18:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=16001], [object_number=44, tag_tid=5, entry_tid=16000], [object_number=47, tag_tid=5, entry_tid=15999], [object_number=46, tag_tid=5, entry_tid=15998]]], state=connected] 2024-08-31T22:18:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=16001], [object_number=44, tag_tid=5, entry_tid=16000], [object_number=47, tag_tid=5, entry_tid=15999], [object_number=46, tag_tid=5, entry_tid=15998]]], state=connected] 2024-08-31T22:18:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=45, tag_tid=5, entry_tid=16001' 2024-08-31T22:18:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=45, tag_tid=5, entry_tid=16001' 2024-08-31T22:18:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=16001' '!=' 'object_number=44, tag_tid=5, entry_tid=15884' 2024-08-31T22:18:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=16001' = 'object_number=44, tag_tid=5, entry_tid=15884' 2024-08-31T22:18:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=45, tag_tid=5, entry_tid=16001' 2024-08-31T22:18:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=16001' 2024-08-31T22:18:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=16001' 2024-08-31T22:18:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16001 2024-08-31T22:18:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 16001 2024-08-31T22:18:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:08.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:08.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:08.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=16117], [object_number=44, tag_tid=5, entry_tid=16116], [object_number=47, tag_tid=5, entry_tid=16115], [object_number=46, tag_tid=5, entry_tid=16114]]], state=connected] 2024-08-31T22:18:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=16117], [object_number=44, tag_tid=5, entry_tid=16116], [object_number=47, tag_tid=5, entry_tid=16115], [object_number=46, tag_tid=5, entry_tid=16114]]], state=connected] 2024-08-31T22:18:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=45, tag_tid=5, entry_tid=16117' 2024-08-31T22:18:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=45, tag_tid=5, entry_tid=16117' 2024-08-31T22:18:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=16117' '!=' 'object_number=45, tag_tid=5, entry_tid=16001' 2024-08-31T22:18:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=16117' = 'object_number=45, tag_tid=5, entry_tid=16001' 2024-08-31T22:18:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=45, tag_tid=5, entry_tid=16117' 2024-08-31T22:18:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=16117' 2024-08-31T22:18:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=16117' 2024-08-31T22:18:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16117 2024-08-31T22:18:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 16117 2024-08-31T22:18:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=16224], [object_number=47, tag_tid=5, entry_tid=16223], [object_number=46, tag_tid=5, entry_tid=16222], [object_number=45, tag_tid=5, entry_tid=16221]]], state=connected] 2024-08-31T22:18:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=16224], [object_number=47, tag_tid=5, entry_tid=16223], [object_number=46, tag_tid=5, entry_tid=16222], [object_number=45, tag_tid=5, entry_tid=16221]]], state=connected] 2024-08-31T22:18:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=16224' 2024-08-31T22:18:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=16224' 2024-08-31T22:18:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=16224' '!=' 'object_number=45, tag_tid=5, entry_tid=16117' 2024-08-31T22:18:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=16224' = 'object_number=45, tag_tid=5, entry_tid=16117' 2024-08-31T22:18:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=16224' 2024-08-31T22:18:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=16224' 2024-08-31T22:18:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=16224' 2024-08-31T22:18:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16224 2024-08-31T22:18:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 16224 2024-08-31T22:18:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:09.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:09.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:09.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:09.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:09.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=16334], [object_number=45, tag_tid=5, entry_tid=16333], [object_number=44, tag_tid=5, entry_tid=16332], [object_number=47, tag_tid=5, entry_tid=16331]]], state=connected] 2024-08-31T22:18:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:09.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:09.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=16334], [object_number=45, tag_tid=5, entry_tid=16333], [object_number=44, tag_tid=5, entry_tid=16332], [object_number=47, tag_tid=5, entry_tid=16331]]], state=connected] 2024-08-31T22:18:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=16334' 2024-08-31T22:18:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=16334' 2024-08-31T22:18:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=16334' '!=' 'object_number=44, tag_tid=5, entry_tid=16224' 2024-08-31T22:18:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=16334' = 'object_number=44, tag_tid=5, entry_tid=16224' 2024-08-31T22:18:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=16334' 2024-08-31T22:18:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=16334' 2024-08-31T22:18:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=16334' 2024-08-31T22:18:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16334 2024-08-31T22:18:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 16334 2024-08-31T22:18:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=16436], [object_number=47, tag_tid=5, entry_tid=16435], [object_number=46, tag_tid=5, entry_tid=16434], [object_number=45, tag_tid=5, entry_tid=16433]]], state=connected] 2024-08-31T22:18:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=16436], [object_number=47, tag_tid=5, entry_tid=16435], [object_number=46, tag_tid=5, entry_tid=16434], [object_number=45, tag_tid=5, entry_tid=16433]]], state=connected] 2024-08-31T22:18:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=16436' 2024-08-31T22:18:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=16436' 2024-08-31T22:18:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=16436' '!=' 'object_number=46, tag_tid=5, entry_tid=16334' 2024-08-31T22:18:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=16436' = 'object_number=46, tag_tid=5, entry_tid=16334' 2024-08-31T22:18:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=16436' 2024-08-31T22:18:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=16436' 2024-08-31T22:18:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=16436' 2024-08-31T22:18:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16436 2024-08-31T22:18:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 16436 2024-08-31T22:18:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=16536], [object_number=47, tag_tid=5, entry_tid=16535], [object_number=46, tag_tid=5, entry_tid=16534], [object_number=45, tag_tid=5, entry_tid=16533]]], state=connected] 2024-08-31T22:18:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=16536], [object_number=47, tag_tid=5, entry_tid=16535], [object_number=46, tag_tid=5, entry_tid=16534], [object_number=45, tag_tid=5, entry_tid=16533]]], state=connected] 2024-08-31T22:18:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=16536' 2024-08-31T22:18:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=16536' 2024-08-31T22:18:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=16536' '!=' 'object_number=44, tag_tid=5, entry_tid=16436' 2024-08-31T22:18:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=16536' = 'object_number=44, tag_tid=5, entry_tid=16436' 2024-08-31T22:18:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=16536' 2024-08-31T22:18:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=16536' 2024-08-31T22:18:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=16536' 2024-08-31T22:18:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16536 2024-08-31T22:18:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 16536 2024-08-31T22:18:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:10.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:10.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:10.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:10.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=16641], [object_number=44, tag_tid=5, entry_tid=16640], [object_number=47, tag_tid=5, entry_tid=16639], [object_number=46, tag_tid=5, entry_tid=16638]]], state=connected] 2024-08-31T22:18:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=16641], [object_number=44, tag_tid=5, entry_tid=16640], [object_number=47, tag_tid=5, entry_tid=16639], [object_number=46, tag_tid=5, entry_tid=16638]]], state=connected] 2024-08-31T22:18:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=45, tag_tid=5, entry_tid=16641' 2024-08-31T22:18:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=45, tag_tid=5, entry_tid=16641' 2024-08-31T22:18:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=16641' '!=' 'object_number=44, tag_tid=5, entry_tid=16536' 2024-08-31T22:18:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=16641' = 'object_number=44, tag_tid=5, entry_tid=16536' 2024-08-31T22:18:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=45, tag_tid=5, entry_tid=16641' 2024-08-31T22:18:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:10.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:10.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=16641' 2024-08-31T22:18:10.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=16641' 2024-08-31T22:18:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16641 2024-08-31T22:18:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 16641 2024-08-31T22:18:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=16745], [object_number=44, tag_tid=5, entry_tid=16744], [object_number=47, tag_tid=5, entry_tid=16743], [object_number=46, tag_tid=5, entry_tid=16742]]], state=connected] 2024-08-31T22:18:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=16745], [object_number=44, tag_tid=5, entry_tid=16744], [object_number=47, tag_tid=5, entry_tid=16743], [object_number=46, tag_tid=5, entry_tid=16742]]], state=connected] 2024-08-31T22:18:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=45, tag_tid=5, entry_tid=16745' 2024-08-31T22:18:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=45, tag_tid=5, entry_tid=16745' 2024-08-31T22:18:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=16745' '!=' 'object_number=45, tag_tid=5, entry_tid=16641' 2024-08-31T22:18:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=16745' = 'object_number=45, tag_tid=5, entry_tid=16641' 2024-08-31T22:18:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=45, tag_tid=5, entry_tid=16745' 2024-08-31T22:18:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=16745' 2024-08-31T22:18:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=16745' 2024-08-31T22:18:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16745 2024-08-31T22:18:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 16745 2024-08-31T22:18:11.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:11.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:11.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:11.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:11.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=16854], [object_number=45, tag_tid=5, entry_tid=16853], [object_number=44, tag_tid=5, entry_tid=16852], [object_number=47, tag_tid=5, entry_tid=16851]]], state=connected] 2024-08-31T22:18:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:11.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:11.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:11.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:11.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:11.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=16854], [object_number=45, tag_tid=5, entry_tid=16853], [object_number=44, tag_tid=5, entry_tid=16852], [object_number=47, tag_tid=5, entry_tid=16851]]], state=connected] 2024-08-31T22:18:11.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=16854' 2024-08-31T22:18:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=16854' 2024-08-31T22:18:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=16854' '!=' 'object_number=45, tag_tid=5, entry_tid=16745' 2024-08-31T22:18:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=16854' = 'object_number=45, tag_tid=5, entry_tid=16745' 2024-08-31T22:18:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=16854' 2024-08-31T22:18:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=16854' 2024-08-31T22:18:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=16854' 2024-08-31T22:18:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16854 2024-08-31T22:18:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 16854 2024-08-31T22:18:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=16970], [object_number=45, tag_tid=5, entry_tid=16969], [object_number=44, tag_tid=5, entry_tid=16968], [object_number=47, tag_tid=5, entry_tid=16967]]], state=connected] 2024-08-31T22:18:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=16970], [object_number=45, tag_tid=5, entry_tid=16969], [object_number=44, tag_tid=5, entry_tid=16968], [object_number=47, tag_tid=5, entry_tid=16967]]], state=connected] 2024-08-31T22:18:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=16970' 2024-08-31T22:18:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=16970' 2024-08-31T22:18:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=16970' '!=' 'object_number=46, tag_tid=5, entry_tid=16854' 2024-08-31T22:18:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=16970' = 'object_number=46, tag_tid=5, entry_tid=16854' 2024-08-31T22:18:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=16970' 2024-08-31T22:18:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=16970' 2024-08-31T22:18:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=16970' 2024-08-31T22:18:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16970 2024-08-31T22:18:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 16970 2024-08-31T22:18:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:12.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:12.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:12.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=17087], [object_number=46, tag_tid=5, entry_tid=17086], [object_number=45, tag_tid=5, entry_tid=17085], [object_number=44, tag_tid=5, entry_tid=17084]]], state=connected] 2024-08-31T22:18:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:12.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:12.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:12.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:12.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:12.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=17087], [object_number=46, tag_tid=5, entry_tid=17086], [object_number=45, tag_tid=5, entry_tid=17085], [object_number=44, tag_tid=5, entry_tid=17084]]], state=connected] 2024-08-31T22:18:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=47, tag_tid=5, entry_tid=17087' 2024-08-31T22:18:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=47, tag_tid=5, entry_tid=17087' 2024-08-31T22:18:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=17087' '!=' 'object_number=46, tag_tid=5, entry_tid=16970' 2024-08-31T22:18:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=17087' = 'object_number=46, tag_tid=5, entry_tid=16970' 2024-08-31T22:18:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=47, tag_tid=5, entry_tid=17087' 2024-08-31T22:18:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=17087' 2024-08-31T22:18:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=17087' 2024-08-31T22:18:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17087 2024-08-31T22:18:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 17087 2024-08-31T22:18:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:12.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=17209], [object_number=44, tag_tid=5, entry_tid=17208], [object_number=47, tag_tid=5, entry_tid=17207], [object_number=46, tag_tid=5, entry_tid=17206]]], state=connected] 2024-08-31T22:18:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=17209], [object_number=44, tag_tid=5, entry_tid=17208], [object_number=47, tag_tid=5, entry_tid=17207], [object_number=46, tag_tid=5, entry_tid=17206]]], state=connected] 2024-08-31T22:18:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=45, tag_tid=5, entry_tid=17209' 2024-08-31T22:18:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=45, tag_tid=5, entry_tid=17209' 2024-08-31T22:18:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=17209' '!=' 'object_number=47, tag_tid=5, entry_tid=17087' 2024-08-31T22:18:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=17209' = 'object_number=47, tag_tid=5, entry_tid=17087' 2024-08-31T22:18:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=45, tag_tid=5, entry_tid=17209' 2024-08-31T22:18:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=17209' 2024-08-31T22:18:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=17209' 2024-08-31T22:18:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17209 2024-08-31T22:18:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 17209 2024-08-31T22:18:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=17314], [object_number=45, tag_tid=5, entry_tid=17313], [object_number=44, tag_tid=5, entry_tid=17312], [object_number=47, tag_tid=5, entry_tid=17311]]], state=connected] 2024-08-31T22:18:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:13.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:13.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:13.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:13.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:13.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:13.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=17314], [object_number=45, tag_tid=5, entry_tid=17313], [object_number=44, tag_tid=5, entry_tid=17312], [object_number=47, tag_tid=5, entry_tid=17311]]], state=connected] 2024-08-31T22:18:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=17314' 2024-08-31T22:18:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=17314' 2024-08-31T22:18:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=17314' '!=' 'object_number=45, tag_tid=5, entry_tid=17209' 2024-08-31T22:18:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=17314' = 'object_number=45, tag_tid=5, entry_tid=17209' 2024-08-31T22:18:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=17314' 2024-08-31T22:18:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=17314' 2024-08-31T22:18:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=17314' 2024-08-31T22:18:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17314 2024-08-31T22:18:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 17314 2024-08-31T22:18:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=17419], [object_number=46, tag_tid=5, entry_tid=17418], [object_number=45, tag_tid=5, entry_tid=17417], [object_number=44, tag_tid=5, entry_tid=17416]]], state=connected] 2024-08-31T22:18:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 11 2024-08-31T22:18:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=17419], [object_number=46, tag_tid=5, entry_tid=17418], [object_number=45, tag_tid=5, entry_tid=17417], [object_number=44, tag_tid=5, entry_tid=17416]]], state=connected] 2024-08-31T22:18:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=47, tag_tid=5, entry_tid=17419' 2024-08-31T22:18:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=47, tag_tid=5, entry_tid=17419' 2024-08-31T22:18:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=17419' '!=' 'object_number=46, tag_tid=5, entry_tid=17314' 2024-08-31T22:18:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=17419' = 'object_number=46, tag_tid=5, entry_tid=17314' 2024-08-31T22:18:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=47, tag_tid=5, entry_tid=17419' 2024-08-31T22:18:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=17419' 2024-08-31T22:18:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:14.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:14.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=17419' 2024-08-31T22:18:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17419 2024-08-31T22:18:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 17419 2024-08-31T22:18:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 12 2024-08-31T22:18:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=50, tag_tid=5, entry_tid=17514], [object_number=49, tag_tid=5, entry_tid=17513], [object_number=48, tag_tid=5, entry_tid=17512], [object_number=51, tag_tid=5, entry_tid=17511]]], state=connected] 2024-08-31T22:18:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 12 2024-08-31T22:18:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=50, tag_tid=5, entry_tid=17514], [object_number=49, tag_tid=5, entry_tid=17513], [object_number=48, tag_tid=5, entry_tid=17512], [object_number=51, tag_tid=5, entry_tid=17511]]], state=connected] 2024-08-31T22:18:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=50, tag_tid=5, entry_tid=17514' 2024-08-31T22:18:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=50, tag_tid=5, entry_tid=17514' 2024-08-31T22:18:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=50, tag_tid=5, entry_tid=17514' '!=' 'object_number=47, tag_tid=5, entry_tid=17419' 2024-08-31T22:18:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=50, tag_tid=5, entry_tid=17514' = 'object_number=47, tag_tid=5, entry_tid=17419' 2024-08-31T22:18:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=50, tag_tid=5, entry_tid=17514' 2024-08-31T22:18:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=50, tag_tid=5, entry_tid=17514' 2024-08-31T22:18:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=50, tag_tid=5, entry_tid=17514' 2024-08-31T22:18:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:14.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17514 2024-08-31T22:18:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 17514 2024-08-31T22:18:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 12 2024-08-31T22:18:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=48, tag_tid=5, entry_tid=17624], [object_number=51, tag_tid=5, entry_tid=17623], [object_number=50, tag_tid=5, entry_tid=17622], [object_number=49, tag_tid=5, entry_tid=17621]]], state=connected] 2024-08-31T22:18:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 12 2024-08-31T22:18:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=48, tag_tid=5, entry_tid=17624], [object_number=51, tag_tid=5, entry_tid=17623], [object_number=50, tag_tid=5, entry_tid=17622], [object_number=49, tag_tid=5, entry_tid=17621]]], state=connected] 2024-08-31T22:18:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=48, tag_tid=5, entry_tid=17624' 2024-08-31T22:18:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=48, tag_tid=5, entry_tid=17624' 2024-08-31T22:18:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=48, tag_tid=5, entry_tid=17624' '!=' 'object_number=50, tag_tid=5, entry_tid=17514' 2024-08-31T22:18:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=48, tag_tid=5, entry_tid=17624' = 'object_number=50, tag_tid=5, entry_tid=17514' 2024-08-31T22:18:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=48, tag_tid=5, entry_tid=17624' 2024-08-31T22:18:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=48, tag_tid=5, entry_tid=17624' 2024-08-31T22:18:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=48, tag_tid=5, entry_tid=17624' 2024-08-31T22:18:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17624 2024-08-31T22:18:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 17624 2024-08-31T22:18:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:15.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:15.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:15.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:15.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:15.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:15.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:15.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:15.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:15.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:15.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:15.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 12 2024-08-31T22:18:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=50, tag_tid=5, entry_tid=17730], [object_number=49, tag_tid=5, entry_tid=17729], [object_number=48, tag_tid=5, entry_tid=17728], [object_number=51, tag_tid=5, entry_tid=17727]]], state=connected] 2024-08-31T22:18:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:15.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:15.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 12 2024-08-31T22:18:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=50, tag_tid=5, entry_tid=17730], [object_number=49, tag_tid=5, entry_tid=17729], [object_number=48, tag_tid=5, entry_tid=17728], [object_number=51, tag_tid=5, entry_tid=17727]]], state=connected] 2024-08-31T22:18:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=50, tag_tid=5, entry_tid=17730' 2024-08-31T22:18:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=50, tag_tid=5, entry_tid=17730' 2024-08-31T22:18:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=50, tag_tid=5, entry_tid=17730' '!=' 'object_number=48, tag_tid=5, entry_tid=17624' 2024-08-31T22:18:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=50, tag_tid=5, entry_tid=17730' = 'object_number=48, tag_tid=5, entry_tid=17624' 2024-08-31T22:18:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=50, tag_tid=5, entry_tid=17730' 2024-08-31T22:18:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=50, tag_tid=5, entry_tid=17730' 2024-08-31T22:18:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=50, tag_tid=5, entry_tid=17730' 2024-08-31T22:18:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17730 2024-08-31T22:18:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 17730 2024-08-31T22:18:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 12 2024-08-31T22:18:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=51, tag_tid=5, entry_tid=17843], [object_number=50, tag_tid=5, entry_tid=17842], [object_number=49, tag_tid=5, entry_tid=17841], [object_number=48, tag_tid=5, entry_tid=17840]]], state=connected] 2024-08-31T22:18:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:16.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 12 2024-08-31T22:18:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=51, tag_tid=5, entry_tid=17843], [object_number=50, tag_tid=5, entry_tid=17842], [object_number=49, tag_tid=5, entry_tid=17841], [object_number=48, tag_tid=5, entry_tid=17840]]], state=connected] 2024-08-31T22:18:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=51, tag_tid=5, entry_tid=17843' 2024-08-31T22:18:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=51, tag_tid=5, entry_tid=17843' 2024-08-31T22:18:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=51, tag_tid=5, entry_tid=17843' '!=' 'object_number=50, tag_tid=5, entry_tid=17730' 2024-08-31T22:18:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=51, tag_tid=5, entry_tid=17843' = 'object_number=50, tag_tid=5, entry_tid=17730' 2024-08-31T22:18:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=51, tag_tid=5, entry_tid=17843' 2024-08-31T22:18:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=51, tag_tid=5, entry_tid=17843' 2024-08-31T22:18:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=51, tag_tid=5, entry_tid=17843' 2024-08-31T22:18:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17843 2024-08-31T22:18:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 17843 2024-08-31T22:18:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:16.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:16.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:16.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:16.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:16.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 12 2024-08-31T22:18:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=50, tag_tid=5, entry_tid=17958], [object_number=49, tag_tid=5, entry_tid=17957], [object_number=48, tag_tid=5, entry_tid=17956], [object_number=51, tag_tid=5, entry_tid=17955]]], state=connected] 2024-08-31T22:18:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 12 2024-08-31T22:18:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=50, tag_tid=5, entry_tid=17958], [object_number=49, tag_tid=5, entry_tid=17957], [object_number=48, tag_tid=5, entry_tid=17956], [object_number=51, tag_tid=5, entry_tid=17955]]], state=connected] 2024-08-31T22:18:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=50, tag_tid=5, entry_tid=17958' 2024-08-31T22:18:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=50, tag_tid=5, entry_tid=17958' 2024-08-31T22:18:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=50, tag_tid=5, entry_tid=17958' '!=' 'object_number=51, tag_tid=5, entry_tid=17843' 2024-08-31T22:18:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=50, tag_tid=5, entry_tid=17958' = 'object_number=51, tag_tid=5, entry_tid=17843' 2024-08-31T22:18:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=50, tag_tid=5, entry_tid=17958' 2024-08-31T22:18:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=50, tag_tid=5, entry_tid=17958' 2024-08-31T22:18:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:16.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:16.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=50, tag_tid=5, entry_tid=17958' 2024-08-31T22:18:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17958 2024-08-31T22:18:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 17958 2024-08-31T22:18:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:16.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:16.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 12 2024-08-31T22:18:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=48, tag_tid=5, entry_tid=17972], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970], [object_number=49, tag_tid=5, entry_tid=17969]]], state=connected] 2024-08-31T22:18:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 12 2024-08-31T22:18:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=48, tag_tid=5, entry_tid=17972], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970], [object_number=49, tag_tid=5, entry_tid=17969]]], state=connected] 2024-08-31T22:18:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=48, tag_tid=5, entry_tid=17972' 2024-08-31T22:18:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=48, tag_tid=5, entry_tid=17972' 2024-08-31T22:18:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=48, tag_tid=5, entry_tid=17972' '!=' 'object_number=50, tag_tid=5, entry_tid=17958' 2024-08-31T22:18:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:18:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=48, tag_tid=5, entry_tid=17972' = 'object_number=50, tag_tid=5, entry_tid=17958' 2024-08-31T22:18:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=48, tag_tid=5, entry_tid=17972' 2024-08-31T22:18:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=6 2024-08-31T22:18:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=48, tag_tid=5, entry_tid=17972' 2024-08-31T22:18:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:18:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=5 2024-08-31T22:18:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:18:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=48, tag_tid=5, entry_tid=17972' 2024-08-31T22:18:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:18:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:18:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17972 2024-08-31T22:18:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 6 = 5 -a 1 -le 17972 2024-08-31T22:18:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:18:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:18:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 12 2024-08-31T22:18:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=48, tag_tid=5, entry_tid=17972], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970], [object_number=49, tag_tid=5, entry_tid=17969]]], state=connected] 2024-08-31T22:18:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 12 2024-08-31T22:18:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=48, tag_tid=5, entry_tid=17972], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970], [object_number=49, tag_tid=5, entry_tid=17969]]], state=connected] 2024-08-31T22:18:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=48, tag_tid=5, entry_tid=17972' 2024-08-31T22:18:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=48, tag_tid=5, entry_tid=17972' 2024-08-31T22:18:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=48, tag_tid=5, entry_tid=17972' '!=' 'object_number=48, tag_tid=5, entry_tid=17972' 2024-08-31T22:18:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:18:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.4 2024-08-31T22:18:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:18:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:18:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:18:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:18:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:18:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:18:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:18:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:18:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:18:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:18:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:18:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:18:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:18:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:18:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:18:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:18:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:18:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:18:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:18:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:18:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:18:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:18:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:18:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:18:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:18:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:18:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:18:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:18:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:18:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:18:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:18:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:18:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:18:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:18:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:18:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:18:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:18:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:18:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:18:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:18:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:18:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:18:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:18:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:18:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:18:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 12 2024-08-31T22:18:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:18:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:18:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:18:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:18:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:18:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:18:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:18:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:18:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:18:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:18:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 12 2024-08-31T22:18:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 12 2024-08-31T22:18:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:18:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=49, tag_tid=6, entry_tid=1], [object_number=48, tag_tid=6, entry_tid=0], [object_number=51, tag_tid=5, entry_tid=17971], [object_number=50, tag_tid=5, entry_tid=17970]]], state=connected] 2024-08-31T22:18:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:18:18.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:18.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=49, tag_tid=6, entry_tid=1' -a 'object_number=49, tag_tid=6, entry_tid=1' = 'object_number=49, tag_tid=6, entry_tid=1' 2024-08-31T22:18:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-08-31T22:18:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap2 2024-08-31T22:18:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:18:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:18:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:18:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-08-31T22:18:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'test': 2024-08-31T22:18:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:18:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:18:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 3 2024-08-31T22:18:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a87b4c6a1b 2024-08-31T22:18:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a87b4c6a1b 2024-08-31T22:18:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:18:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:18:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:18:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:18:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:14:57 2024 2024-08-31T22:18:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:16:45 2024 2024-08-31T22:18:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:18:00 2024 2024-08-31T22:18:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:18:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a87b4c6a1b 2024-08-31T22:18:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:18:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:18:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:18:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:18:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n snap1-clean ']' 2024-08-31T22:18:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap1-clean 2024-08-31T22:18:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap1-clean 2024-08-31T22:18:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:18:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:18:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:18:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:18:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:18:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:18:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:18:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:18:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:18:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:18:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap2 2024-08-31T22:18:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap2 2024-08-31T22:18:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:18:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:18:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:18:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:18:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:18:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:18:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:18:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:18:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:18:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ clean_snap_name=snap2-clean 2024-08-31T22:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-08-31T22:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap2-clean 2024-08-31T22:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-08-31T22:18:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:18:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror test 2024-08-31T22:18:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:18:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:18:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=12 2024-08-31T22:18:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:18:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-08-31T22:18:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 132999168 bytes 2024-08-31T22:18:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 188816384 bytes 2024-08-31T22:18:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 222733312 bytes 2024-08-31T22:18:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 278108160 bytes 2024-08-31T22:18:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:500 IOs, 324495360 bytes 2024-08-31T22:18:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:18:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:18:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:18:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-08-31T22:18:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:18:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:18:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:18:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:18:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:18:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-08-31T22:18:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:18:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:18:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:18:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:18:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:18:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-08-31T22:18:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='test: 2024-08-31T22:18:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: global_id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:18:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+replaying 2024-08-31T22:18:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: replaying, {"bytes_per_second":3184892.9,"entries_behind_primary":0,"entries_per_second":199.27,"non_primary_position":{"entry_tid":1,"object_number":49,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":49,"tag_tid":6}} 2024-08-31T22:18:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: service: mirror.0 on smithi032 2024-08-31T22:18:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:18:25 2024-08-31T22:18:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: peer_sites: 2024-08-31T22:18:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: name: cluster2 2024-08-31T22:18:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+stopped 2024-08-31T22:18:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: local image is primary 2024-08-31T22:18:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:18:35' 2024-08-31T22:18:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ state: .*up+replaying' 2024-08-31T22:18:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: state: up+replaying 2024-08-31T22:18:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ description: .*' 2024-08-31T22:18:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: description: replaying, {"bytes_per_second":3184892.9,"entries_behind_primary":0,"entries_per_second":199.27,"non_primary_position":{"entry_tid":1,"object_number":49,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":49,"tag_tid":6}} 2024-08-31T22:18:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:18:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:18:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^up+' 2024-08-31T22:18:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:18:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep 'service: *mirror..* on ' 2024-08-31T22:18:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: service: mirror.0 on smithi032 2024-08-31T22:18:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update 2024-08-31T22:18:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-08-31T22:18:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:18:25' 2024-08-31T22:18:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-08-31 22:18:25' 2024-08-31T22:18:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:18:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:18:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'prev_last_update=2024-08-31 22:18:25' 2024-08-31T22:18:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:18:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:18:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-08-31T22:18:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='OKOKOK142574264mirror.0smithi03219.1.1-244-g3091dfc0trueOK42614267mirror.1smithi03219.1.1-244-g3091dfc0falseOKtestdd2c705e-25cf-4565-981f-d48de48eee84up+replayingreplaying, {"bytes_per_second":3184892.9,"entries_behind_primary":0,"entries_per_second":199.27,"non_primary_position":{"entry_tid":1,"object_number":49,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":49,"tag_tid":6}}42574264mirror.0smithi0322024-08-31 22:18:25cluster24ca32b00-2164-486e-837e-e93795a16af1up+stoppedlocal image is primary2024-08-31 22:18:35' 2024-08-31T22:18:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update state 2024-08-31T22:18:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-08-31T22:18:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:18:25' 2024-08-31T22:18:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-08-31T22:18:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ state=up+replaying 2024-08-31T22:18:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:18:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep up+replaying 2024-08-31T22:18:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:18:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:18:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap3 2024-08-31T22:18:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap3 2024-08-31T22:18:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:18:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:18:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:18:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap3 2024-08-31T22:18:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:19:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-08-31T22:19:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-08-31T22:19:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:19:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:19:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:19:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror test started 2024-08-31T22:19:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:19:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:19:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:19:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-08-31T22:19:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:19:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:19:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:19:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:19:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:19:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:19:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:19:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-08-31T22:19:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:19:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:19:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:19:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:19:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:19:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:19:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:19:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:19:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:19:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:19:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:19:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:19:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:19:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:19:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:19:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:19:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-08-31T22:19:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:19:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:19:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:19:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:19:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:19:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:19:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:19:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:19:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:19:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:19:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:19:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:19:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:19:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:19:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:19:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:19:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:10.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:10.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:10.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:10.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:10.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:10.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:10.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:10.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:10.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:10.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=8385], [object_number=56, tag_tid=8, entry_tid=8384], [object_number=59, tag_tid=8, entry_tid=8383], [object_number=58, tag_tid=8, entry_tid=8382]]], state=connected] 2024-08-31T22:19:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=8385], [object_number=56, tag_tid=8, entry_tid=8384], [object_number=59, tag_tid=8, entry_tid=8383], [object_number=58, tag_tid=8, entry_tid=8382]]], state=connected] 2024-08-31T22:19:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=57, tag_tid=8, entry_tid=8385' 2024-08-31T22:19:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=57, tag_tid=8, entry_tid=8385' 2024-08-31T22:19:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=8385' '!=' '' 2024-08-31T22:19:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=8385' = '' 2024-08-31T22:19:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=57, tag_tid=8, entry_tid=8385' 2024-08-31T22:19:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=8385' 2024-08-31T22:19:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:10.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:10.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=8385' 2024-08-31T22:19:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8385 2024-08-31T22:19:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 8385 2024-08-31T22:19:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:11.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:11.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:11.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:11.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:11.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:11.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:11.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:11.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:11.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:11.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:11.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:11.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=8556], [object_number=59, tag_tid=8, entry_tid=8555], [object_number=58, tag_tid=8, entry_tid=8554], [object_number=57, tag_tid=8, entry_tid=8553]]], state=connected] 2024-08-31T22:19:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=8556], [object_number=59, tag_tid=8, entry_tid=8555], [object_number=58, tag_tid=8, entry_tid=8554], [object_number=57, tag_tid=8, entry_tid=8553]]], state=connected] 2024-08-31T22:19:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=56, tag_tid=8, entry_tid=8556' 2024-08-31T22:19:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=56, tag_tid=8, entry_tid=8556' 2024-08-31T22:19:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=8556' '!=' 'object_number=57, tag_tid=8, entry_tid=8385' 2024-08-31T22:19:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=8556' = 'object_number=57, tag_tid=8, entry_tid=8385' 2024-08-31T22:19:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=56, tag_tid=8, entry_tid=8556' 2024-08-31T22:19:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:11.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:11.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=8556' 2024-08-31T22:19:11.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:11.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:11.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=8556' 2024-08-31T22:19:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8556 2024-08-31T22:19:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 8556 2024-08-31T22:19:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:11.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:11.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:11.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:11.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:11.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:11.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:11.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:11.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=8678], [object_number=57, tag_tid=8, entry_tid=8677], [object_number=56, tag_tid=8, entry_tid=8676], [object_number=59, tag_tid=8, entry_tid=8675]]], state=connected] 2024-08-31T22:19:11.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:11.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:11.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:11.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:11.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:11.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:11.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=8678], [object_number=57, tag_tid=8, entry_tid=8677], [object_number=56, tag_tid=8, entry_tid=8676], [object_number=59, tag_tid=8, entry_tid=8675]]], state=connected] 2024-08-31T22:19:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=58, tag_tid=8, entry_tid=8678' 2024-08-31T22:19:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=58, tag_tid=8, entry_tid=8678' 2024-08-31T22:19:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=8678' '!=' 'object_number=56, tag_tid=8, entry_tid=8556' 2024-08-31T22:19:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=8678' = 'object_number=56, tag_tid=8, entry_tid=8556' 2024-08-31T22:19:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=58, tag_tid=8, entry_tid=8678' 2024-08-31T22:19:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=8678' 2024-08-31T22:19:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=8678' 2024-08-31T22:19:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8678 2024-08-31T22:19:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 8678 2024-08-31T22:19:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:11.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:11.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:11.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=8797], [object_number=56, tag_tid=8, entry_tid=8796], [object_number=59, tag_tid=8, entry_tid=8795], [object_number=58, tag_tid=8, entry_tid=8794]]], state=connected] 2024-08-31T22:19:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=8797], [object_number=56, tag_tid=8, entry_tid=8796], [object_number=59, tag_tid=8, entry_tid=8795], [object_number=58, tag_tid=8, entry_tid=8794]]], state=connected] 2024-08-31T22:19:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=57, tag_tid=8, entry_tid=8797' 2024-08-31T22:19:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=57, tag_tid=8, entry_tid=8797' 2024-08-31T22:19:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=8797' '!=' 'object_number=58, tag_tid=8, entry_tid=8678' 2024-08-31T22:19:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=8797' = 'object_number=58, tag_tid=8, entry_tid=8678' 2024-08-31T22:19:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=57, tag_tid=8, entry_tid=8797' 2024-08-31T22:19:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=8797' 2024-08-31T22:19:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=8797' 2024-08-31T22:19:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8797 2024-08-31T22:19:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 8797 2024-08-31T22:19:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=8919], [object_number=58, tag_tid=8, entry_tid=8918], [object_number=57, tag_tid=8, entry_tid=8917], [object_number=56, tag_tid=8, entry_tid=8916]]], state=connected] 2024-08-31T22:19:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=8919], [object_number=58, tag_tid=8, entry_tid=8918], [object_number=57, tag_tid=8, entry_tid=8917], [object_number=56, tag_tid=8, entry_tid=8916]]], state=connected] 2024-08-31T22:19:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=59, tag_tid=8, entry_tid=8919' 2024-08-31T22:19:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=59, tag_tid=8, entry_tid=8919' 2024-08-31T22:19:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=8919' '!=' 'object_number=57, tag_tid=8, entry_tid=8797' 2024-08-31T22:19:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=8919' = 'object_number=57, tag_tid=8, entry_tid=8797' 2024-08-31T22:19:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=59, tag_tid=8, entry_tid=8919' 2024-08-31T22:19:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=8919' 2024-08-31T22:19:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=8919' 2024-08-31T22:19:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8919 2024-08-31T22:19:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 8919 2024-08-31T22:19:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:13.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:13.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:13.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:13.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:13.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:13.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:13.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:13.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:13.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=9046], [object_number=57, tag_tid=8, entry_tid=9045], [object_number=56, tag_tid=8, entry_tid=9044], [object_number=59, tag_tid=8, entry_tid=9043]]], state=connected] 2024-08-31T22:19:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=9046], [object_number=57, tag_tid=8, entry_tid=9045], [object_number=56, tag_tid=8, entry_tid=9044], [object_number=59, tag_tid=8, entry_tid=9043]]], state=connected] 2024-08-31T22:19:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=58, tag_tid=8, entry_tid=9046' 2024-08-31T22:19:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=58, tag_tid=8, entry_tid=9046' 2024-08-31T22:19:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=9046' '!=' 'object_number=59, tag_tid=8, entry_tid=8919' 2024-08-31T22:19:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=9046' = 'object_number=59, tag_tid=8, entry_tid=8919' 2024-08-31T22:19:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=58, tag_tid=8, entry_tid=9046' 2024-08-31T22:19:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=9046' 2024-08-31T22:19:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=9046' 2024-08-31T22:19:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9046 2024-08-31T22:19:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 9046 2024-08-31T22:19:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=9156], [object_number=59, tag_tid=8, entry_tid=9155], [object_number=58, tag_tid=8, entry_tid=9154], [object_number=57, tag_tid=8, entry_tid=9153]]], state=connected] 2024-08-31T22:19:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=9156], [object_number=59, tag_tid=8, entry_tid=9155], [object_number=58, tag_tid=8, entry_tid=9154], [object_number=57, tag_tid=8, entry_tid=9153]]], state=connected] 2024-08-31T22:19:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=56, tag_tid=8, entry_tid=9156' 2024-08-31T22:19:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=56, tag_tid=8, entry_tid=9156' 2024-08-31T22:19:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=9156' '!=' 'object_number=58, tag_tid=8, entry_tid=9046' 2024-08-31T22:19:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=9156' = 'object_number=58, tag_tid=8, entry_tid=9046' 2024-08-31T22:19:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=56, tag_tid=8, entry_tid=9156' 2024-08-31T22:19:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=9156' 2024-08-31T22:19:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=9156' 2024-08-31T22:19:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9156 2024-08-31T22:19:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 9156 2024-08-31T22:19:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=9262], [object_number=57, tag_tid=8, entry_tid=9261], [object_number=56, tag_tid=8, entry_tid=9260], [object_number=59, tag_tid=8, entry_tid=9259]]], state=connected] 2024-08-31T22:19:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=9262], [object_number=57, tag_tid=8, entry_tid=9261], [object_number=56, tag_tid=8, entry_tid=9260], [object_number=59, tag_tid=8, entry_tid=9259]]], state=connected] 2024-08-31T22:19:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=58, tag_tid=8, entry_tid=9262' 2024-08-31T22:19:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=58, tag_tid=8, entry_tid=9262' 2024-08-31T22:19:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=9262' '!=' 'object_number=56, tag_tid=8, entry_tid=9156' 2024-08-31T22:19:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=9262' = 'object_number=56, tag_tid=8, entry_tid=9156' 2024-08-31T22:19:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=58, tag_tid=8, entry_tid=9262' 2024-08-31T22:19:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=9262' 2024-08-31T22:19:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=9262' 2024-08-31T22:19:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9262 2024-08-31T22:19:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 9262 2024-08-31T22:19:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:14.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:14.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:14.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=9368], [object_number=59, tag_tid=8, entry_tid=9367], [object_number=58, tag_tid=8, entry_tid=9366], [object_number=57, tag_tid=8, entry_tid=9365]]], state=connected] 2024-08-31T22:19:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=9368], [object_number=59, tag_tid=8, entry_tid=9367], [object_number=58, tag_tid=8, entry_tid=9366], [object_number=57, tag_tid=8, entry_tid=9365]]], state=connected] 2024-08-31T22:19:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=56, tag_tid=8, entry_tid=9368' 2024-08-31T22:19:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=56, tag_tid=8, entry_tid=9368' 2024-08-31T22:19:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=9368' '!=' 'object_number=58, tag_tid=8, entry_tid=9262' 2024-08-31T22:19:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=9368' = 'object_number=58, tag_tid=8, entry_tid=9262' 2024-08-31T22:19:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=56, tag_tid=8, entry_tid=9368' 2024-08-31T22:19:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=9368' 2024-08-31T22:19:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=9368' 2024-08-31T22:19:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9368 2024-08-31T22:19:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 9368 2024-08-31T22:19:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=9475], [object_number=58, tag_tid=8, entry_tid=9474], [object_number=57, tag_tid=8, entry_tid=9473], [object_number=56, tag_tid=8, entry_tid=9472]]], state=connected] 2024-08-31T22:19:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=9475], [object_number=58, tag_tid=8, entry_tid=9474], [object_number=57, tag_tid=8, entry_tid=9473], [object_number=56, tag_tid=8, entry_tid=9472]]], state=connected] 2024-08-31T22:19:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=59, tag_tid=8, entry_tid=9475' 2024-08-31T22:19:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=59, tag_tid=8, entry_tid=9475' 2024-08-31T22:19:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=9475' '!=' 'object_number=56, tag_tid=8, entry_tid=9368' 2024-08-31T22:19:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=9475' = 'object_number=56, tag_tid=8, entry_tid=9368' 2024-08-31T22:19:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=59, tag_tid=8, entry_tid=9475' 2024-08-31T22:19:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=9475' 2024-08-31T22:19:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=9475' 2024-08-31T22:19:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9475 2024-08-31T22:19:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 9475 2024-08-31T22:19:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:15.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:15.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:15.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:15.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:15.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:15.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:15.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:15.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:15.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:15.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:15.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:15.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=9572], [object_number=59, tag_tid=8, entry_tid=9571], [object_number=58, tag_tid=8, entry_tid=9570], [object_number=57, tag_tid=8, entry_tid=9569]]], state=connected] 2024-08-31T22:19:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=9572], [object_number=59, tag_tid=8, entry_tid=9571], [object_number=58, tag_tid=8, entry_tid=9570], [object_number=57, tag_tid=8, entry_tid=9569]]], state=connected] 2024-08-31T22:19:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=56, tag_tid=8, entry_tid=9572' 2024-08-31T22:19:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=56, tag_tid=8, entry_tid=9572' 2024-08-31T22:19:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=9572' '!=' 'object_number=59, tag_tid=8, entry_tid=9475' 2024-08-31T22:19:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=9572' = 'object_number=59, tag_tid=8, entry_tid=9475' 2024-08-31T22:19:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=56, tag_tid=8, entry_tid=9572' 2024-08-31T22:19:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=9572' 2024-08-31T22:19:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=9572' 2024-08-31T22:19:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9572 2024-08-31T22:19:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 9572 2024-08-31T22:19:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=9658], [object_number=57, tag_tid=8, entry_tid=9657], [object_number=56, tag_tid=8, entry_tid=9656], [object_number=59, tag_tid=8, entry_tid=9655]]], state=connected] 2024-08-31T22:19:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:16.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:16.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=9658], [object_number=57, tag_tid=8, entry_tid=9657], [object_number=56, tag_tid=8, entry_tid=9656], [object_number=59, tag_tid=8, entry_tid=9655]]], state=connected] 2024-08-31T22:19:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=58, tag_tid=8, entry_tid=9658' 2024-08-31T22:19:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=58, tag_tid=8, entry_tid=9658' 2024-08-31T22:19:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=9658' '!=' 'object_number=56, tag_tid=8, entry_tid=9572' 2024-08-31T22:19:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=9658' = 'object_number=56, tag_tid=8, entry_tid=9572' 2024-08-31T22:19:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=58, tag_tid=8, entry_tid=9658' 2024-08-31T22:19:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=9658' 2024-08-31T22:19:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=9658' 2024-08-31T22:19:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9658 2024-08-31T22:19:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 9658 2024-08-31T22:19:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=9758], [object_number=57, tag_tid=8, entry_tid=9757], [object_number=56, tag_tid=8, entry_tid=9756], [object_number=59, tag_tid=8, entry_tid=9755]]], state=connected] 2024-08-31T22:19:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=9758], [object_number=57, tag_tid=8, entry_tid=9757], [object_number=56, tag_tid=8, entry_tid=9756], [object_number=59, tag_tid=8, entry_tid=9755]]], state=connected] 2024-08-31T22:19:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=58, tag_tid=8, entry_tid=9758' 2024-08-31T22:19:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=58, tag_tid=8, entry_tid=9758' 2024-08-31T22:19:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=9758' '!=' 'object_number=58, tag_tid=8, entry_tid=9658' 2024-08-31T22:19:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=9758' = 'object_number=58, tag_tid=8, entry_tid=9658' 2024-08-31T22:19:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=58, tag_tid=8, entry_tid=9758' 2024-08-31T22:19:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=9758' 2024-08-31T22:19:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:16.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=9758' 2024-08-31T22:19:16.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:16.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9758 2024-08-31T22:19:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 9758 2024-08-31T22:19:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=9871], [object_number=58, tag_tid=8, entry_tid=9870], [object_number=57, tag_tid=8, entry_tid=9869], [object_number=56, tag_tid=8, entry_tid=9868]]], state=connected] 2024-08-31T22:19:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:17.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:17.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:17.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:17.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:17.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=9871], [object_number=58, tag_tid=8, entry_tid=9870], [object_number=57, tag_tid=8, entry_tid=9869], [object_number=56, tag_tid=8, entry_tid=9868]]], state=connected] 2024-08-31T22:19:17.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=59, tag_tid=8, entry_tid=9871' 2024-08-31T22:19:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=59, tag_tid=8, entry_tid=9871' 2024-08-31T22:19:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=9871' '!=' 'object_number=58, tag_tid=8, entry_tid=9758' 2024-08-31T22:19:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=9871' = 'object_number=58, tag_tid=8, entry_tid=9758' 2024-08-31T22:19:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=59, tag_tid=8, entry_tid=9871' 2024-08-31T22:19:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:17.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=9871' 2024-08-31T22:19:17.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:17.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=9871' 2024-08-31T22:19:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9871 2024-08-31T22:19:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 9871 2024-08-31T22:19:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=63, tag_tid=8, entry_tid=9927], [object_number=58, tag_tid=8, entry_tid=9926], [object_number=57, tag_tid=8, entry_tid=9925], [object_number=56, tag_tid=8, entry_tid=9924]]], state=connected] 2024-08-31T22:19:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 14 2024-08-31T22:19:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=63, tag_tid=8, entry_tid=9927], [object_number=58, tag_tid=8, entry_tid=9926], [object_number=57, tag_tid=8, entry_tid=9925], [object_number=56, tag_tid=8, entry_tid=9924]]], state=connected] 2024-08-31T22:19:17.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=63, tag_tid=8, entry_tid=9927' 2024-08-31T22:19:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=63, tag_tid=8, entry_tid=9927' 2024-08-31T22:19:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=63, tag_tid=8, entry_tid=9927' '!=' 'object_number=59, tag_tid=8, entry_tid=9871' 2024-08-31T22:19:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=63, tag_tid=8, entry_tid=9927' = 'object_number=59, tag_tid=8, entry_tid=9871' 2024-08-31T22:19:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=63, tag_tid=8, entry_tid=9927' 2024-08-31T22:19:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:17.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:17.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:17.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=63, tag_tid=8, entry_tid=9927' 2024-08-31T22:19:17.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:17.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:17.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=63, tag_tid=8, entry_tid=9927' 2024-08-31T22:19:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9927 2024-08-31T22:19:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 9927 2024-08-31T22:19:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:18.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:18.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=60, tag_tid=8, entry_tid=9996], [object_number=63, tag_tid=8, entry_tid=9995], [object_number=62, tag_tid=8, entry_tid=9994], [object_number=61, tag_tid=8, entry_tid=9993]]], state=connected] 2024-08-31T22:19:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:18.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=60, tag_tid=8, entry_tid=9996], [object_number=63, tag_tid=8, entry_tid=9995], [object_number=62, tag_tid=8, entry_tid=9994], [object_number=61, tag_tid=8, entry_tid=9993]]], state=connected] 2024-08-31T22:19:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=60, tag_tid=8, entry_tid=9996' 2024-08-31T22:19:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=60, tag_tid=8, entry_tid=9996' 2024-08-31T22:19:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=60, tag_tid=8, entry_tid=9996' '!=' 'object_number=63, tag_tid=8, entry_tid=9927' 2024-08-31T22:19:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=60, tag_tid=8, entry_tid=9996' = 'object_number=63, tag_tid=8, entry_tid=9927' 2024-08-31T22:19:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=60, tag_tid=8, entry_tid=9996' 2024-08-31T22:19:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=60, tag_tid=8, entry_tid=9996' 2024-08-31T22:19:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=60, tag_tid=8, entry_tid=9996' 2024-08-31T22:19:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9996 2024-08-31T22:19:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 9996 2024-08-31T22:19:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:18.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:18.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:18.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:18.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:18.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:18.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=10122], [object_number=61, tag_tid=8, entry_tid=10121], [object_number=60, tag_tid=8, entry_tid=10120], [object_number=63, tag_tid=8, entry_tid=10119]]], state=connected] 2024-08-31T22:19:18.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:18.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:18.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=10122], [object_number=61, tag_tid=8, entry_tid=10121], [object_number=60, tag_tid=8, entry_tid=10120], [object_number=63, tag_tid=8, entry_tid=10119]]], state=connected] 2024-08-31T22:19:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=62, tag_tid=8, entry_tid=10122' 2024-08-31T22:19:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=62, tag_tid=8, entry_tid=10122' 2024-08-31T22:19:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=10122' '!=' 'object_number=60, tag_tid=8, entry_tid=9996' 2024-08-31T22:19:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=10122' = 'object_number=60, tag_tid=8, entry_tid=9996' 2024-08-31T22:19:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=62, tag_tid=8, entry_tid=10122' 2024-08-31T22:19:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=10122' 2024-08-31T22:19:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=10122' 2024-08-31T22:19:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10122 2024-08-31T22:19:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 10122 2024-08-31T22:19:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=10226], [object_number=61, tag_tid=8, entry_tid=10225], [object_number=60, tag_tid=8, entry_tid=10224], [object_number=63, tag_tid=8, entry_tid=10223]]], state=connected] 2024-08-31T22:19:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=10226], [object_number=61, tag_tid=8, entry_tid=10225], [object_number=60, tag_tid=8, entry_tid=10224], [object_number=63, tag_tid=8, entry_tid=10223]]], state=connected] 2024-08-31T22:19:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=62, tag_tid=8, entry_tid=10226' 2024-08-31T22:19:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=62, tag_tid=8, entry_tid=10226' 2024-08-31T22:19:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=10226' '!=' 'object_number=62, tag_tid=8, entry_tid=10122' 2024-08-31T22:19:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=10226' = 'object_number=62, tag_tid=8, entry_tid=10122' 2024-08-31T22:19:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=62, tag_tid=8, entry_tid=10226' 2024-08-31T22:19:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=10226' 2024-08-31T22:19:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:19.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:19.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=10226' 2024-08-31T22:19:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10226 2024-08-31T22:19:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 10226 2024-08-31T22:19:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=60, tag_tid=8, entry_tid=10332], [object_number=63, tag_tid=8, entry_tid=10331], [object_number=62, tag_tid=8, entry_tid=10330], [object_number=61, tag_tid=8, entry_tid=10329]]], state=connected] 2024-08-31T22:19:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=60, tag_tid=8, entry_tid=10332], [object_number=63, tag_tid=8, entry_tid=10331], [object_number=62, tag_tid=8, entry_tid=10330], [object_number=61, tag_tid=8, entry_tid=10329]]], state=connected] 2024-08-31T22:19:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=60, tag_tid=8, entry_tid=10332' 2024-08-31T22:19:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=60, tag_tid=8, entry_tid=10332' 2024-08-31T22:19:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=60, tag_tid=8, entry_tid=10332' '!=' 'object_number=62, tag_tid=8, entry_tid=10226' 2024-08-31T22:19:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=60, tag_tid=8, entry_tid=10332' = 'object_number=62, tag_tid=8, entry_tid=10226' 2024-08-31T22:19:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=60, tag_tid=8, entry_tid=10332' 2024-08-31T22:19:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=60, tag_tid=8, entry_tid=10332' 2024-08-31T22:19:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=60, tag_tid=8, entry_tid=10332' 2024-08-31T22:19:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10332 2024-08-31T22:19:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 10332 2024-08-31T22:19:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:20.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:20.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:20.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=63, tag_tid=8, entry_tid=10455], [object_number=62, tag_tid=8, entry_tid=10454], [object_number=61, tag_tid=8, entry_tid=10453], [object_number=60, tag_tid=8, entry_tid=10452]]], state=connected] 2024-08-31T22:19:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=63, tag_tid=8, entry_tid=10455], [object_number=62, tag_tid=8, entry_tid=10454], [object_number=61, tag_tid=8, entry_tid=10453], [object_number=60, tag_tid=8, entry_tid=10452]]], state=connected] 2024-08-31T22:19:20.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:20.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=63, tag_tid=8, entry_tid=10455' 2024-08-31T22:19:20.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=63, tag_tid=8, entry_tid=10455' 2024-08-31T22:19:20.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=63, tag_tid=8, entry_tid=10455' '!=' 'object_number=60, tag_tid=8, entry_tid=10332' 2024-08-31T22:19:20.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=63, tag_tid=8, entry_tid=10455' = 'object_number=60, tag_tid=8, entry_tid=10332' 2024-08-31T22:19:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=63, tag_tid=8, entry_tid=10455' 2024-08-31T22:19:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:20.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=63, tag_tid=8, entry_tid=10455' 2024-08-31T22:19:20.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:20.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:20.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=63, tag_tid=8, entry_tid=10455' 2024-08-31T22:19:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10455 2024-08-31T22:19:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 10455 2024-08-31T22:19:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:20.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:20.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:20.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=63, tag_tid=8, entry_tid=10607], [object_number=62, tag_tid=8, entry_tid=10606], [object_number=61, tag_tid=8, entry_tid=10605], [object_number=60, tag_tid=8, entry_tid=10604]]], state=connected] 2024-08-31T22:19:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=63, tag_tid=8, entry_tid=10607], [object_number=62, tag_tid=8, entry_tid=10606], [object_number=61, tag_tid=8, entry_tid=10605], [object_number=60, tag_tid=8, entry_tid=10604]]], state=connected] 2024-08-31T22:19:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=63, tag_tid=8, entry_tid=10607' 2024-08-31T22:19:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=63, tag_tid=8, entry_tid=10607' 2024-08-31T22:19:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=63, tag_tid=8, entry_tid=10607' '!=' 'object_number=63, tag_tid=8, entry_tid=10455' 2024-08-31T22:19:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=63, tag_tid=8, entry_tid=10607' = 'object_number=63, tag_tid=8, entry_tid=10455' 2024-08-31T22:19:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=63, tag_tid=8, entry_tid=10607' 2024-08-31T22:19:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=63, tag_tid=8, entry_tid=10607' 2024-08-31T22:19:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=63, tag_tid=8, entry_tid=10607' 2024-08-31T22:19:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10607 2024-08-31T22:19:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 10607 2024-08-31T22:19:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=60, tag_tid=8, entry_tid=10708], [object_number=63, tag_tid=8, entry_tid=10707], [object_number=62, tag_tid=8, entry_tid=10706], [object_number=61, tag_tid=8, entry_tid=10705]]], state=connected] 2024-08-31T22:19:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=60, tag_tid=8, entry_tid=10708], [object_number=63, tag_tid=8, entry_tid=10707], [object_number=62, tag_tid=8, entry_tid=10706], [object_number=61, tag_tid=8, entry_tid=10705]]], state=connected] 2024-08-31T22:19:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=60, tag_tid=8, entry_tid=10708' 2024-08-31T22:19:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=60, tag_tid=8, entry_tid=10708' 2024-08-31T22:19:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=60, tag_tid=8, entry_tid=10708' '!=' 'object_number=63, tag_tid=8, entry_tid=10607' 2024-08-31T22:19:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=60, tag_tid=8, entry_tid=10708' = 'object_number=63, tag_tid=8, entry_tid=10607' 2024-08-31T22:19:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=60, tag_tid=8, entry_tid=10708' 2024-08-31T22:19:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=60, tag_tid=8, entry_tid=10708' 2024-08-31T22:19:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=60, tag_tid=8, entry_tid=10708' 2024-08-31T22:19:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10708 2024-08-31T22:19:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 10708 2024-08-31T22:19:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:21.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=61, tag_tid=8, entry_tid=10825], [object_number=60, tag_tid=8, entry_tid=10824], [object_number=63, tag_tid=8, entry_tid=10823], [object_number=62, tag_tid=8, entry_tid=10822]]], state=connected] 2024-08-31T22:19:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=61, tag_tid=8, entry_tid=10825], [object_number=60, tag_tid=8, entry_tid=10824], [object_number=63, tag_tid=8, entry_tid=10823], [object_number=62, tag_tid=8, entry_tid=10822]]], state=connected] 2024-08-31T22:19:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=61, tag_tid=8, entry_tid=10825' 2024-08-31T22:19:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=61, tag_tid=8, entry_tid=10825' 2024-08-31T22:19:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=61, tag_tid=8, entry_tid=10825' '!=' 'object_number=60, tag_tid=8, entry_tid=10708' 2024-08-31T22:19:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=61, tag_tid=8, entry_tid=10825' = 'object_number=60, tag_tid=8, entry_tid=10708' 2024-08-31T22:19:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=61, tag_tid=8, entry_tid=10825' 2024-08-31T22:19:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:21.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:21.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=61, tag_tid=8, entry_tid=10825' 2024-08-31T22:19:21.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:21.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=61, tag_tid=8, entry_tid=10825' 2024-08-31T22:19:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10825 2024-08-31T22:19:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 10825 2024-08-31T22:19:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:22.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:22.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:22.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:22.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=60, tag_tid=8, entry_tid=10932], [object_number=63, tag_tid=8, entry_tid=10931], [object_number=62, tag_tid=8, entry_tid=10930], [object_number=61, tag_tid=8, entry_tid=10929]]], state=connected] 2024-08-31T22:19:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=60, tag_tid=8, entry_tid=10932], [object_number=63, tag_tid=8, entry_tid=10931], [object_number=62, tag_tid=8, entry_tid=10930], [object_number=61, tag_tid=8, entry_tid=10929]]], state=connected] 2024-08-31T22:19:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=60, tag_tid=8, entry_tid=10932' 2024-08-31T22:19:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=60, tag_tid=8, entry_tid=10932' 2024-08-31T22:19:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=60, tag_tid=8, entry_tid=10932' '!=' 'object_number=61, tag_tid=8, entry_tid=10825' 2024-08-31T22:19:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=60, tag_tid=8, entry_tid=10932' = 'object_number=61, tag_tid=8, entry_tid=10825' 2024-08-31T22:19:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=60, tag_tid=8, entry_tid=10932' 2024-08-31T22:19:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=60, tag_tid=8, entry_tid=10932' 2024-08-31T22:19:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:22.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:22.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=60, tag_tid=8, entry_tid=10932' 2024-08-31T22:19:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10932 2024-08-31T22:19:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 10932 2024-08-31T22:19:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=63, tag_tid=8, entry_tid=10999], [object_number=62, tag_tid=8, entry_tid=10998], [object_number=61, tag_tid=8, entry_tid=10997], [object_number=60, tag_tid=8, entry_tid=10996]]], state=connected] 2024-08-31T22:19:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=63, tag_tid=8, entry_tid=10999], [object_number=62, tag_tid=8, entry_tid=10998], [object_number=61, tag_tid=8, entry_tid=10997], [object_number=60, tag_tid=8, entry_tid=10996]]], state=connected] 2024-08-31T22:19:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=63, tag_tid=8, entry_tid=10999' 2024-08-31T22:19:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=63, tag_tid=8, entry_tid=10999' 2024-08-31T22:19:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=63, tag_tid=8, entry_tid=10999' '!=' 'object_number=60, tag_tid=8, entry_tid=10932' 2024-08-31T22:19:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=63, tag_tid=8, entry_tid=10999' = 'object_number=60, tag_tid=8, entry_tid=10932' 2024-08-31T22:19:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=63, tag_tid=8, entry_tid=10999' 2024-08-31T22:19:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=63, tag_tid=8, entry_tid=10999' 2024-08-31T22:19:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=63, tag_tid=8, entry_tid=10999' 2024-08-31T22:19:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10999 2024-08-31T22:19:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 10999 2024-08-31T22:19:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=61, tag_tid=8, entry_tid=11097], [object_number=60, tag_tid=8, entry_tid=11096], [object_number=63, tag_tid=8, entry_tid=11095], [object_number=62, tag_tid=8, entry_tid=11094]]], state=connected] 2024-08-31T22:19:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=61, tag_tid=8, entry_tid=11097], [object_number=60, tag_tid=8, entry_tid=11096], [object_number=63, tag_tid=8, entry_tid=11095], [object_number=62, tag_tid=8, entry_tid=11094]]], state=connected] 2024-08-31T22:19:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=61, tag_tid=8, entry_tid=11097' 2024-08-31T22:19:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=61, tag_tid=8, entry_tid=11097' 2024-08-31T22:19:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=61, tag_tid=8, entry_tid=11097' '!=' 'object_number=63, tag_tid=8, entry_tid=10999' 2024-08-31T22:19:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=61, tag_tid=8, entry_tid=11097' = 'object_number=63, tag_tid=8, entry_tid=10999' 2024-08-31T22:19:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=61, tag_tid=8, entry_tid=11097' 2024-08-31T22:19:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=61, tag_tid=8, entry_tid=11097' 2024-08-31T22:19:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=61, tag_tid=8, entry_tid=11097' 2024-08-31T22:19:23.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:23.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11097 2024-08-31T22:19:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 11097 2024-08-31T22:19:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=11218], [object_number=61, tag_tid=8, entry_tid=11217], [object_number=60, tag_tid=8, entry_tid=11216], [object_number=63, tag_tid=8, entry_tid=11215]]], state=connected] 2024-08-31T22:19:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=11218], [object_number=61, tag_tid=8, entry_tid=11217], [object_number=60, tag_tid=8, entry_tid=11216], [object_number=63, tag_tid=8, entry_tid=11215]]], state=connected] 2024-08-31T22:19:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=62, tag_tid=8, entry_tid=11218' 2024-08-31T22:19:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=62, tag_tid=8, entry_tid=11218' 2024-08-31T22:19:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=11218' '!=' 'object_number=61, tag_tid=8, entry_tid=11097' 2024-08-31T22:19:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=11218' = 'object_number=61, tag_tid=8, entry_tid=11097' 2024-08-31T22:19:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=62, tag_tid=8, entry_tid=11218' 2024-08-31T22:19:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=11218' 2024-08-31T22:19:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=11218' 2024-08-31T22:19:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11218 2024-08-31T22:19:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 11218 2024-08-31T22:19:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=11326], [object_number=61, tag_tid=8, entry_tid=11325], [object_number=60, tag_tid=8, entry_tid=11324], [object_number=63, tag_tid=8, entry_tid=11323]]], state=connected] 2024-08-31T22:19:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=11326], [object_number=61, tag_tid=8, entry_tid=11325], [object_number=60, tag_tid=8, entry_tid=11324], [object_number=63, tag_tid=8, entry_tid=11323]]], state=connected] 2024-08-31T22:19:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=62, tag_tid=8, entry_tid=11326' 2024-08-31T22:19:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=62, tag_tid=8, entry_tid=11326' 2024-08-31T22:19:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=11326' '!=' 'object_number=62, tag_tid=8, entry_tid=11218' 2024-08-31T22:19:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=11326' = 'object_number=62, tag_tid=8, entry_tid=11218' 2024-08-31T22:19:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=62, tag_tid=8, entry_tid=11326' 2024-08-31T22:19:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=11326' 2024-08-31T22:19:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=11326' 2024-08-31T22:19:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:24.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11326 2024-08-31T22:19:24.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 11326 2024-08-31T22:19:24.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:24.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:24.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:24.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:24.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:24.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:24.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:24.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:24.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:24.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:24.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:24.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=11438], [object_number=61, tag_tid=8, entry_tid=11437], [object_number=60, tag_tid=8, entry_tid=11436], [object_number=63, tag_tid=8, entry_tid=11435]]], state=connected] 2024-08-31T22:19:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=11438], [object_number=61, tag_tid=8, entry_tid=11437], [object_number=60, tag_tid=8, entry_tid=11436], [object_number=63, tag_tid=8, entry_tid=11435]]], state=connected] 2024-08-31T22:19:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=62, tag_tid=8, entry_tid=11438' 2024-08-31T22:19:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=62, tag_tid=8, entry_tid=11438' 2024-08-31T22:19:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=11438' '!=' 'object_number=62, tag_tid=8, entry_tid=11326' 2024-08-31T22:19:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=11438' = 'object_number=62, tag_tid=8, entry_tid=11326' 2024-08-31T22:19:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=62, tag_tid=8, entry_tid=11438' 2024-08-31T22:19:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=11438' 2024-08-31T22:19:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=11438' 2024-08-31T22:19:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11438 2024-08-31T22:19:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 11438 2024-08-31T22:19:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=61, tag_tid=8, entry_tid=11541], [object_number=60, tag_tid=8, entry_tid=11540], [object_number=63, tag_tid=8, entry_tid=11539], [object_number=62, tag_tid=8, entry_tid=11538]]], state=connected] 2024-08-31T22:19:25.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:25.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=61, tag_tid=8, entry_tid=11541], [object_number=60, tag_tid=8, entry_tid=11540], [object_number=63, tag_tid=8, entry_tid=11539], [object_number=62, tag_tid=8, entry_tid=11538]]], state=connected] 2024-08-31T22:19:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=61, tag_tid=8, entry_tid=11541' 2024-08-31T22:19:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=61, tag_tid=8, entry_tid=11541' 2024-08-31T22:19:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=61, tag_tid=8, entry_tid=11541' '!=' 'object_number=62, tag_tid=8, entry_tid=11438' 2024-08-31T22:19:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=61, tag_tid=8, entry_tid=11541' = 'object_number=62, tag_tid=8, entry_tid=11438' 2024-08-31T22:19:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=61, tag_tid=8, entry_tid=11541' 2024-08-31T22:19:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=61, tag_tid=8, entry_tid=11541' 2024-08-31T22:19:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=61, tag_tid=8, entry_tid=11541' 2024-08-31T22:19:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11541 2024-08-31T22:19:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 11541 2024-08-31T22:19:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=61, tag_tid=8, entry_tid=11641], [object_number=60, tag_tid=8, entry_tid=11640], [object_number=63, tag_tid=8, entry_tid=11639], [object_number=62, tag_tid=8, entry_tid=11638]]], state=connected] 2024-08-31T22:19:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=61, tag_tid=8, entry_tid=11641], [object_number=60, tag_tid=8, entry_tid=11640], [object_number=63, tag_tid=8, entry_tid=11639], [object_number=62, tag_tid=8, entry_tid=11638]]], state=connected] 2024-08-31T22:19:25.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=61, tag_tid=8, entry_tid=11641' 2024-08-31T22:19:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=61, tag_tid=8, entry_tid=11641' 2024-08-31T22:19:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=61, tag_tid=8, entry_tid=11641' '!=' 'object_number=61, tag_tid=8, entry_tid=11541' 2024-08-31T22:19:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=61, tag_tid=8, entry_tid=11641' = 'object_number=61, tag_tid=8, entry_tid=11541' 2024-08-31T22:19:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=61, tag_tid=8, entry_tid=11641' 2024-08-31T22:19:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=61, tag_tid=8, entry_tid=11641' 2024-08-31T22:19:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=61, tag_tid=8, entry_tid=11641' 2024-08-31T22:19:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11641 2024-08-31T22:19:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 11641 2024-08-31T22:19:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:26.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=11722], [object_number=61, tag_tid=8, entry_tid=11721], [object_number=60, tag_tid=8, entry_tid=11720], [object_number=63, tag_tid=8, entry_tid=11719]]], state=connected] 2024-08-31T22:19:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:26.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:26.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:26.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:26.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:26.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=11722], [object_number=61, tag_tid=8, entry_tid=11721], [object_number=60, tag_tid=8, entry_tid=11720], [object_number=63, tag_tid=8, entry_tid=11719]]], state=connected] 2024-08-31T22:19:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=62, tag_tid=8, entry_tid=11722' 2024-08-31T22:19:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=62, tag_tid=8, entry_tid=11722' 2024-08-31T22:19:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=11722' '!=' 'object_number=61, tag_tid=8, entry_tid=11641' 2024-08-31T22:19:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=11722' = 'object_number=61, tag_tid=8, entry_tid=11641' 2024-08-31T22:19:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=62, tag_tid=8, entry_tid=11722' 2024-08-31T22:19:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=11722' 2024-08-31T22:19:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=11722' 2024-08-31T22:19:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11722 2024-08-31T22:19:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 11722 2024-08-31T22:19:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=61, tag_tid=8, entry_tid=11833], [object_number=60, tag_tid=8, entry_tid=11832], [object_number=63, tag_tid=8, entry_tid=11831], [object_number=62, tag_tid=8, entry_tid=11830]]], state=connected] 2024-08-31T22:19:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=61, tag_tid=8, entry_tid=11833], [object_number=60, tag_tid=8, entry_tid=11832], [object_number=63, tag_tid=8, entry_tid=11831], [object_number=62, tag_tid=8, entry_tid=11830]]], state=connected] 2024-08-31T22:19:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=61, tag_tid=8, entry_tid=11833' 2024-08-31T22:19:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=61, tag_tid=8, entry_tid=11833' 2024-08-31T22:19:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=61, tag_tid=8, entry_tid=11833' '!=' 'object_number=62, tag_tid=8, entry_tid=11722' 2024-08-31T22:19:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=61, tag_tid=8, entry_tid=11833' = 'object_number=62, tag_tid=8, entry_tid=11722' 2024-08-31T22:19:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=61, tag_tid=8, entry_tid=11833' 2024-08-31T22:19:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=61, tag_tid=8, entry_tid=11833' 2024-08-31T22:19:26.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:26.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:26.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=61, tag_tid=8, entry_tid=11833' 2024-08-31T22:19:26.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:26.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:26.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11833 2024-08-31T22:19:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 11833 2024-08-31T22:19:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:27.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=11934], [object_number=61, tag_tid=8, entry_tid=11933], [object_number=60, tag_tid=8, entry_tid=11932], [object_number=63, tag_tid=8, entry_tid=11931]]], state=connected] 2024-08-31T22:19:27.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=11934], [object_number=61, tag_tid=8, entry_tid=11933], [object_number=60, tag_tid=8, entry_tid=11932], [object_number=63, tag_tid=8, entry_tid=11931]]], state=connected] 2024-08-31T22:19:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=62, tag_tid=8, entry_tid=11934' 2024-08-31T22:19:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=62, tag_tid=8, entry_tid=11934' 2024-08-31T22:19:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=11934' '!=' 'object_number=61, tag_tid=8, entry_tid=11833' 2024-08-31T22:19:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=11934' = 'object_number=61, tag_tid=8, entry_tid=11833' 2024-08-31T22:19:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=62, tag_tid=8, entry_tid=11934' 2024-08-31T22:19:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=11934' 2024-08-31T22:19:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=11934' 2024-08-31T22:19:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11934 2024-08-31T22:19:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 11934 2024-08-31T22:19:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:27.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:27.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:27.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:27.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=61, tag_tid=8, entry_tid=12041], [object_number=60, tag_tid=8, entry_tid=12040], [object_number=63, tag_tid=8, entry_tid=12039], [object_number=62, tag_tid=8, entry_tid=12038]]], state=connected] 2024-08-31T22:19:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:28.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=61, tag_tid=8, entry_tid=12041], [object_number=60, tag_tid=8, entry_tid=12040], [object_number=63, tag_tid=8, entry_tid=12039], [object_number=62, tag_tid=8, entry_tid=12038]]], state=connected] 2024-08-31T22:19:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=61, tag_tid=8, entry_tid=12041' 2024-08-31T22:19:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=61, tag_tid=8, entry_tid=12041' 2024-08-31T22:19:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=61, tag_tid=8, entry_tid=12041' '!=' 'object_number=62, tag_tid=8, entry_tid=11934' 2024-08-31T22:19:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=61, tag_tid=8, entry_tid=12041' = 'object_number=62, tag_tid=8, entry_tid=11934' 2024-08-31T22:19:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=61, tag_tid=8, entry_tid=12041' 2024-08-31T22:19:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=61, tag_tid=8, entry_tid=12041' 2024-08-31T22:19:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=61, tag_tid=8, entry_tid=12041' 2024-08-31T22:19:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12041 2024-08-31T22:19:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 12041 2024-08-31T22:19:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:28.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:28.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:28.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:28.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:28.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:28.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:28.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:28.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:28.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:28.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=60, tag_tid=8, entry_tid=12180], [object_number=63, tag_tid=8, entry_tid=12179], [object_number=62, tag_tid=8, entry_tid=12178], [object_number=61, tag_tid=8, entry_tid=12177]]], state=connected] 2024-08-31T22:19:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:28.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:28.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:28.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:28.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:28.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=60, tag_tid=8, entry_tid=12180], [object_number=63, tag_tid=8, entry_tid=12179], [object_number=62, tag_tid=8, entry_tid=12178], [object_number=61, tag_tid=8, entry_tid=12177]]], state=connected] 2024-08-31T22:19:28.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=60, tag_tid=8, entry_tid=12180' 2024-08-31T22:19:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=60, tag_tid=8, entry_tid=12180' 2024-08-31T22:19:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=60, tag_tid=8, entry_tid=12180' '!=' 'object_number=61, tag_tid=8, entry_tid=12041' 2024-08-31T22:19:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=60, tag_tid=8, entry_tid=12180' = 'object_number=61, tag_tid=8, entry_tid=12041' 2024-08-31T22:19:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=60, tag_tid=8, entry_tid=12180' 2024-08-31T22:19:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=60, tag_tid=8, entry_tid=12180' 2024-08-31T22:19:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=60, tag_tid=8, entry_tid=12180' 2024-08-31T22:19:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12180 2024-08-31T22:19:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 12180 2024-08-31T22:19:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=12274], [object_number=61, tag_tid=8, entry_tid=12273], [object_number=60, tag_tid=8, entry_tid=12272], [object_number=63, tag_tid=8, entry_tid=12271]]], state=connected] 2024-08-31T22:19:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:29.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:29.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:29.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:29.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:29.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=12274], [object_number=61, tag_tid=8, entry_tid=12273], [object_number=60, tag_tid=8, entry_tid=12272], [object_number=63, tag_tid=8, entry_tid=12271]]], state=connected] 2024-08-31T22:19:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=62, tag_tid=8, entry_tid=12274' 2024-08-31T22:19:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=62, tag_tid=8, entry_tid=12274' 2024-08-31T22:19:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=12274' '!=' 'object_number=60, tag_tid=8, entry_tid=12180' 2024-08-31T22:19:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=12274' = 'object_number=60, tag_tid=8, entry_tid=12180' 2024-08-31T22:19:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=62, tag_tid=8, entry_tid=12274' 2024-08-31T22:19:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=12274' 2024-08-31T22:19:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=12274' 2024-08-31T22:19:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12274 2024-08-31T22:19:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 12274 2024-08-31T22:19:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:29.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:29.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:29.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:29.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:29.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:29.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:29.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:29.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=61, tag_tid=8, entry_tid=12369], [object_number=60, tag_tid=8, entry_tid=12368], [object_number=63, tag_tid=8, entry_tid=12367], [object_number=62, tag_tid=8, entry_tid=12366]]], state=connected] 2024-08-31T22:19:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=61, tag_tid=8, entry_tid=12369], [object_number=60, tag_tid=8, entry_tid=12368], [object_number=63, tag_tid=8, entry_tid=12367], [object_number=62, tag_tid=8, entry_tid=12366]]], state=connected] 2024-08-31T22:19:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=61, tag_tid=8, entry_tid=12369' 2024-08-31T22:19:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=61, tag_tid=8, entry_tid=12369' 2024-08-31T22:19:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=61, tag_tid=8, entry_tid=12369' '!=' 'object_number=62, tag_tid=8, entry_tid=12274' 2024-08-31T22:19:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=61, tag_tid=8, entry_tid=12369' = 'object_number=62, tag_tid=8, entry_tid=12274' 2024-08-31T22:19:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=61, tag_tid=8, entry_tid=12369' 2024-08-31T22:19:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=61, tag_tid=8, entry_tid=12369' 2024-08-31T22:19:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=61, tag_tid=8, entry_tid=12369' 2024-08-31T22:19:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12369 2024-08-31T22:19:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 12369 2024-08-31T22:19:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=63, tag_tid=8, entry_tid=12459], [object_number=62, tag_tid=8, entry_tid=12458], [object_number=61, tag_tid=8, entry_tid=12457], [object_number=60, tag_tid=8, entry_tid=12456]]], state=connected] 2024-08-31T22:19:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:29.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:29.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=63, tag_tid=8, entry_tid=12459], [object_number=62, tag_tid=8, entry_tid=12458], [object_number=61, tag_tid=8, entry_tid=12457], [object_number=60, tag_tid=8, entry_tid=12456]]], state=connected] 2024-08-31T22:19:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=63, tag_tid=8, entry_tid=12459' 2024-08-31T22:19:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=63, tag_tid=8, entry_tid=12459' 2024-08-31T22:19:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=63, tag_tid=8, entry_tid=12459' '!=' 'object_number=61, tag_tid=8, entry_tid=12369' 2024-08-31T22:19:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=63, tag_tid=8, entry_tid=12459' = 'object_number=61, tag_tid=8, entry_tid=12369' 2024-08-31T22:19:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=63, tag_tid=8, entry_tid=12459' 2024-08-31T22:19:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=63, tag_tid=8, entry_tid=12459' 2024-08-31T22:19:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=63, tag_tid=8, entry_tid=12459' 2024-08-31T22:19:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12459 2024-08-31T22:19:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 12459 2024-08-31T22:19:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:30.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=12542], [object_number=61, tag_tid=8, entry_tid=12541], [object_number=60, tag_tid=8, entry_tid=12540], [object_number=63, tag_tid=8, entry_tid=12539]]], state=connected] 2024-08-31T22:19:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=12542], [object_number=61, tag_tid=8, entry_tid=12541], [object_number=60, tag_tid=8, entry_tid=12540], [object_number=63, tag_tid=8, entry_tid=12539]]], state=connected] 2024-08-31T22:19:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=62, tag_tid=8, entry_tid=12542' 2024-08-31T22:19:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=62, tag_tid=8, entry_tid=12542' 2024-08-31T22:19:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=12542' '!=' 'object_number=63, tag_tid=8, entry_tid=12459' 2024-08-31T22:19:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=12542' = 'object_number=63, tag_tid=8, entry_tid=12459' 2024-08-31T22:19:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=62, tag_tid=8, entry_tid=12542' 2024-08-31T22:19:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=12542' 2024-08-31T22:19:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=12542' 2024-08-31T22:19:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12542 2024-08-31T22:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 12542 2024-08-31T22:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=12642], [object_number=61, tag_tid=8, entry_tid=12641], [object_number=60, tag_tid=8, entry_tid=12640], [object_number=63, tag_tid=8, entry_tid=12639]]], state=connected] 2024-08-31T22:19:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=12642], [object_number=61, tag_tid=8, entry_tid=12641], [object_number=60, tag_tid=8, entry_tid=12640], [object_number=63, tag_tid=8, entry_tid=12639]]], state=connected] 2024-08-31T22:19:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=62, tag_tid=8, entry_tid=12642' 2024-08-31T22:19:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=62, tag_tid=8, entry_tid=12642' 2024-08-31T22:19:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=12642' '!=' 'object_number=62, tag_tid=8, entry_tid=12542' 2024-08-31T22:19:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=12642' = 'object_number=62, tag_tid=8, entry_tid=12542' 2024-08-31T22:19:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=62, tag_tid=8, entry_tid=12642' 2024-08-31T22:19:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=12642' 2024-08-31T22:19:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=12642' 2024-08-31T22:19:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12642 2024-08-31T22:19:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 12642 2024-08-31T22:19:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:31.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:31.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=63, tag_tid=8, entry_tid=12747], [object_number=62, tag_tid=8, entry_tid=12746], [object_number=61, tag_tid=8, entry_tid=12745], [object_number=60, tag_tid=8, entry_tid=12744]]], state=connected] 2024-08-31T22:19:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=63, tag_tid=8, entry_tid=12747], [object_number=62, tag_tid=8, entry_tid=12746], [object_number=61, tag_tid=8, entry_tid=12745], [object_number=60, tag_tid=8, entry_tid=12744]]], state=connected] 2024-08-31T22:19:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=63, tag_tid=8, entry_tid=12747' 2024-08-31T22:19:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=63, tag_tid=8, entry_tid=12747' 2024-08-31T22:19:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=63, tag_tid=8, entry_tid=12747' '!=' 'object_number=62, tag_tid=8, entry_tid=12642' 2024-08-31T22:19:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=63, tag_tid=8, entry_tid=12747' = 'object_number=62, tag_tid=8, entry_tid=12642' 2024-08-31T22:19:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=63, tag_tid=8, entry_tid=12747' 2024-08-31T22:19:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=63, tag_tid=8, entry_tid=12747' 2024-08-31T22:19:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=63, tag_tid=8, entry_tid=12747' 2024-08-31T22:19:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12747 2024-08-31T22:19:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 12747 2024-08-31T22:19:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=60, tag_tid=8, entry_tid=12840], [object_number=63, tag_tid=8, entry_tid=12839], [object_number=62, tag_tid=8, entry_tid=12838], [object_number=61, tag_tid=8, entry_tid=12837]]], state=connected] 2024-08-31T22:19:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=60, tag_tid=8, entry_tid=12840], [object_number=63, tag_tid=8, entry_tid=12839], [object_number=62, tag_tid=8, entry_tid=12838], [object_number=61, tag_tid=8, entry_tid=12837]]], state=connected] 2024-08-31T22:19:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=60, tag_tid=8, entry_tid=12840' 2024-08-31T22:19:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=60, tag_tid=8, entry_tid=12840' 2024-08-31T22:19:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=60, tag_tid=8, entry_tid=12840' '!=' 'object_number=63, tag_tid=8, entry_tid=12747' 2024-08-31T22:19:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=60, tag_tid=8, entry_tid=12840' = 'object_number=63, tag_tid=8, entry_tid=12747' 2024-08-31T22:19:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=60, tag_tid=8, entry_tid=12840' 2024-08-31T22:19:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=60, tag_tid=8, entry_tid=12840' 2024-08-31T22:19:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=60, tag_tid=8, entry_tid=12840' 2024-08-31T22:19:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12840 2024-08-31T22:19:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 12840 2024-08-31T22:19:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=60, tag_tid=8, entry_tid=12928], [object_number=63, tag_tid=8, entry_tid=12927], [object_number=62, tag_tid=8, entry_tid=12926], [object_number=61, tag_tid=8, entry_tid=12925]]], state=connected] 2024-08-31T22:19:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:32.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:32.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:32.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=60, tag_tid=8, entry_tid=12928], [object_number=63, tag_tid=8, entry_tid=12927], [object_number=62, tag_tid=8, entry_tid=12926], [object_number=61, tag_tid=8, entry_tid=12925]]], state=connected] 2024-08-31T22:19:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=60, tag_tid=8, entry_tid=12928' 2024-08-31T22:19:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=60, tag_tid=8, entry_tid=12928' 2024-08-31T22:19:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=60, tag_tid=8, entry_tid=12928' '!=' 'object_number=60, tag_tid=8, entry_tid=12840' 2024-08-31T22:19:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=60, tag_tid=8, entry_tid=12928' = 'object_number=60, tag_tid=8, entry_tid=12840' 2024-08-31T22:19:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=60, tag_tid=8, entry_tid=12928' 2024-08-31T22:19:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=60, tag_tid=8, entry_tid=12928' 2024-08-31T22:19:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=60, tag_tid=8, entry_tid=12928' 2024-08-31T22:19:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12928 2024-08-31T22:19:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 12928 2024-08-31T22:19:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:32.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:32.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:32.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:32.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:32.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=63, tag_tid=8, entry_tid=13043], [object_number=62, tag_tid=8, entry_tid=13042], [object_number=61, tag_tid=8, entry_tid=13041], [object_number=60, tag_tid=8, entry_tid=13040]]], state=connected] 2024-08-31T22:19:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=63, tag_tid=8, entry_tid=13043], [object_number=62, tag_tid=8, entry_tid=13042], [object_number=61, tag_tid=8, entry_tid=13041], [object_number=60, tag_tid=8, entry_tid=13040]]], state=connected] 2024-08-31T22:19:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=63, tag_tid=8, entry_tid=13043' 2024-08-31T22:19:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=63, tag_tid=8, entry_tid=13043' 2024-08-31T22:19:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=63, tag_tid=8, entry_tid=13043' '!=' 'object_number=60, tag_tid=8, entry_tid=12928' 2024-08-31T22:19:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=63, tag_tid=8, entry_tid=13043' = 'object_number=60, tag_tid=8, entry_tid=12928' 2024-08-31T22:19:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=63, tag_tid=8, entry_tid=13043' 2024-08-31T22:19:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:32.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=63, tag_tid=8, entry_tid=13043' 2024-08-31T22:19:32.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:32.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=63, tag_tid=8, entry_tid=13043' 2024-08-31T22:19:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13043 2024-08-31T22:19:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 13043 2024-08-31T22:19:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=13146], [object_number=61, tag_tid=8, entry_tid=13145], [object_number=60, tag_tid=8, entry_tid=13144], [object_number=63, tag_tid=8, entry_tid=13143]]], state=connected] 2024-08-31T22:19:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=62, tag_tid=8, entry_tid=13146], [object_number=61, tag_tid=8, entry_tid=13145], [object_number=60, tag_tid=8, entry_tid=13144], [object_number=63, tag_tid=8, entry_tid=13143]]], state=connected] 2024-08-31T22:19:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:33.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=62, tag_tid=8, entry_tid=13146' 2024-08-31T22:19:33.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=62, tag_tid=8, entry_tid=13146' 2024-08-31T22:19:33.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=13146' '!=' 'object_number=63, tag_tid=8, entry_tid=13043' 2024-08-31T22:19:33.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:33.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=62, tag_tid=8, entry_tid=13146' = 'object_number=63, tag_tid=8, entry_tid=13043' 2024-08-31T22:19:33.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=62, tag_tid=8, entry_tid=13146' 2024-08-31T22:19:33.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:33.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:33.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=13146' 2024-08-31T22:19:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=62, tag_tid=8, entry_tid=13146' 2024-08-31T22:19:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13146 2024-08-31T22:19:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 13146 2024-08-31T22:19:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:33.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=61, tag_tid=8, entry_tid=13261], [object_number=60, tag_tid=8, entry_tid=13260], [object_number=63, tag_tid=8, entry_tid=13259], [object_number=62, tag_tid=8, entry_tid=13258]]], state=connected] 2024-08-31T22:19:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:33.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:33.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=61, tag_tid=8, entry_tid=13261], [object_number=60, tag_tid=8, entry_tid=13260], [object_number=63, tag_tid=8, entry_tid=13259], [object_number=62, tag_tid=8, entry_tid=13258]]], state=connected] 2024-08-31T22:19:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=61, tag_tid=8, entry_tid=13261' 2024-08-31T22:19:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=61, tag_tid=8, entry_tid=13261' 2024-08-31T22:19:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=61, tag_tid=8, entry_tid=13261' '!=' 'object_number=62, tag_tid=8, entry_tid=13146' 2024-08-31T22:19:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=61, tag_tid=8, entry_tid=13261' = 'object_number=62, tag_tid=8, entry_tid=13146' 2024-08-31T22:19:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=61, tag_tid=8, entry_tid=13261' 2024-08-31T22:19:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=61, tag_tid=8, entry_tid=13261' 2024-08-31T22:19:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=61, tag_tid=8, entry_tid=13261' 2024-08-31T22:19:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13261 2024-08-31T22:19:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 13261 2024-08-31T22:19:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:34.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:34.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:34.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=61, tag_tid=8, entry_tid=13389], [object_number=60, tag_tid=8, entry_tid=13388], [object_number=63, tag_tid=8, entry_tid=13387], [object_number=62, tag_tid=8, entry_tid=13386]]], state=connected] 2024-08-31T22:19:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=61, tag_tid=8, entry_tid=13389], [object_number=60, tag_tid=8, entry_tid=13388], [object_number=63, tag_tid=8, entry_tid=13387], [object_number=62, tag_tid=8, entry_tid=13386]]], state=connected] 2024-08-31T22:19:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=61, tag_tid=8, entry_tid=13389' 2024-08-31T22:19:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=61, tag_tid=8, entry_tid=13389' 2024-08-31T22:19:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=61, tag_tid=8, entry_tid=13389' '!=' 'object_number=61, tag_tid=8, entry_tid=13261' 2024-08-31T22:19:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=61, tag_tid=8, entry_tid=13389' = 'object_number=61, tag_tid=8, entry_tid=13261' 2024-08-31T22:19:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=61, tag_tid=8, entry_tid=13389' 2024-08-31T22:19:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=61, tag_tid=8, entry_tid=13389' 2024-08-31T22:19:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=61, tag_tid=8, entry_tid=13389' 2024-08-31T22:19:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13389 2024-08-31T22:19:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 13389 2024-08-31T22:19:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:34.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:34.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:34.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:34.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:34.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:34.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:34.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=60, tag_tid=8, entry_tid=13472], [object_number=67, tag_tid=8, entry_tid=13471], [object_number=62, tag_tid=8, entry_tid=13470], [object_number=61, tag_tid=8, entry_tid=13469]]], state=connected] 2024-08-31T22:19:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 15 2024-08-31T22:19:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=60, tag_tid=8, entry_tid=13472], [object_number=67, tag_tid=8, entry_tid=13471], [object_number=62, tag_tid=8, entry_tid=13470], [object_number=61, tag_tid=8, entry_tid=13469]]], state=connected] 2024-08-31T22:19:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=60, tag_tid=8, entry_tid=13472' 2024-08-31T22:19:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=60, tag_tid=8, entry_tid=13472' 2024-08-31T22:19:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=60, tag_tid=8, entry_tid=13472' '!=' 'object_number=61, tag_tid=8, entry_tid=13389' 2024-08-31T22:19:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=60, tag_tid=8, entry_tid=13472' = 'object_number=61, tag_tid=8, entry_tid=13389' 2024-08-31T22:19:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=60, tag_tid=8, entry_tid=13472' 2024-08-31T22:19:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=60, tag_tid=8, entry_tid=13472' 2024-08-31T22:19:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=60, tag_tid=8, entry_tid=13472' 2024-08-31T22:19:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13472 2024-08-31T22:19:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 13472 2024-08-31T22:19:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:35.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:35.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:35.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:35.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:35.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:35.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:35.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:35.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:35.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=67, tag_tid=8, entry_tid=13555], [object_number=66, tag_tid=8, entry_tid=13554], [object_number=65, tag_tid=8, entry_tid=13553], [object_number=64, tag_tid=8, entry_tid=13552]]], state=connected] 2024-08-31T22:19:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:35.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:35.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:35.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=67, tag_tid=8, entry_tid=13555], [object_number=66, tag_tid=8, entry_tid=13554], [object_number=65, tag_tid=8, entry_tid=13553], [object_number=64, tag_tid=8, entry_tid=13552]]], state=connected] 2024-08-31T22:19:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=67, tag_tid=8, entry_tid=13555' 2024-08-31T22:19:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=67, tag_tid=8, entry_tid=13555' 2024-08-31T22:19:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=67, tag_tid=8, entry_tid=13555' '!=' 'object_number=60, tag_tid=8, entry_tid=13472' 2024-08-31T22:19:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=67, tag_tid=8, entry_tid=13555' = 'object_number=60, tag_tid=8, entry_tid=13472' 2024-08-31T22:19:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=67, tag_tid=8, entry_tid=13555' 2024-08-31T22:19:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=67, tag_tid=8, entry_tid=13555' 2024-08-31T22:19:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=67, tag_tid=8, entry_tid=13555' 2024-08-31T22:19:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13555 2024-08-31T22:19:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 13555 2024-08-31T22:19:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=65, tag_tid=8, entry_tid=13677], [object_number=64, tag_tid=8, entry_tid=13676], [object_number=67, tag_tid=8, entry_tid=13675], [object_number=66, tag_tid=8, entry_tid=13674]]], state=connected] 2024-08-31T22:19:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=65, tag_tid=8, entry_tid=13677], [object_number=64, tag_tid=8, entry_tid=13676], [object_number=67, tag_tid=8, entry_tid=13675], [object_number=66, tag_tid=8, entry_tid=13674]]], state=connected] 2024-08-31T22:19:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=65, tag_tid=8, entry_tid=13677' 2024-08-31T22:19:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=65, tag_tid=8, entry_tid=13677' 2024-08-31T22:19:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=65, tag_tid=8, entry_tid=13677' '!=' 'object_number=67, tag_tid=8, entry_tid=13555' 2024-08-31T22:19:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=65, tag_tid=8, entry_tid=13677' = 'object_number=67, tag_tid=8, entry_tid=13555' 2024-08-31T22:19:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=65, tag_tid=8, entry_tid=13677' 2024-08-31T22:19:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=65, tag_tid=8, entry_tid=13677' 2024-08-31T22:19:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=65, tag_tid=8, entry_tid=13677' 2024-08-31T22:19:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13677 2024-08-31T22:19:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 13677 2024-08-31T22:19:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=65, tag_tid=8, entry_tid=13785], [object_number=64, tag_tid=8, entry_tid=13784], [object_number=67, tag_tid=8, entry_tid=13783], [object_number=66, tag_tid=8, entry_tid=13782]]], state=connected] 2024-08-31T22:19:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=65, tag_tid=8, entry_tid=13785], [object_number=64, tag_tid=8, entry_tid=13784], [object_number=67, tag_tid=8, entry_tid=13783], [object_number=66, tag_tid=8, entry_tid=13782]]], state=connected] 2024-08-31T22:19:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=65, tag_tid=8, entry_tid=13785' 2024-08-31T22:19:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=65, tag_tid=8, entry_tid=13785' 2024-08-31T22:19:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=65, tag_tid=8, entry_tid=13785' '!=' 'object_number=65, tag_tid=8, entry_tid=13677' 2024-08-31T22:19:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=65, tag_tid=8, entry_tid=13785' = 'object_number=65, tag_tid=8, entry_tid=13677' 2024-08-31T22:19:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=65, tag_tid=8, entry_tid=13785' 2024-08-31T22:19:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=65, tag_tid=8, entry_tid=13785' 2024-08-31T22:19:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=65, tag_tid=8, entry_tid=13785' 2024-08-31T22:19:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13785 2024-08-31T22:19:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 13785 2024-08-31T22:19:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=65, tag_tid=8, entry_tid=13869], [object_number=64, tag_tid=8, entry_tid=13868], [object_number=67, tag_tid=8, entry_tid=13867], [object_number=66, tag_tid=8, entry_tid=13866]]], state=connected] 2024-08-31T22:19:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=65, tag_tid=8, entry_tid=13869], [object_number=64, tag_tid=8, entry_tid=13868], [object_number=67, tag_tid=8, entry_tid=13867], [object_number=66, tag_tid=8, entry_tid=13866]]], state=connected] 2024-08-31T22:19:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=65, tag_tid=8, entry_tid=13869' 2024-08-31T22:19:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=65, tag_tid=8, entry_tid=13869' 2024-08-31T22:19:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=65, tag_tid=8, entry_tid=13869' '!=' 'object_number=65, tag_tid=8, entry_tid=13785' 2024-08-31T22:19:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=65, tag_tid=8, entry_tid=13869' = 'object_number=65, tag_tid=8, entry_tid=13785' 2024-08-31T22:19:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=65, tag_tid=8, entry_tid=13869' 2024-08-31T22:19:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=65, tag_tid=8, entry_tid=13869' 2024-08-31T22:19:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=65, tag_tid=8, entry_tid=13869' 2024-08-31T22:19:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13869 2024-08-31T22:19:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 13869 2024-08-31T22:19:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:37.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:37.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:37.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:37.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:37.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:37.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=13974], [object_number=65, tag_tid=8, entry_tid=13973], [object_number=64, tag_tid=8, entry_tid=13972], [object_number=67, tag_tid=8, entry_tid=13971]]], state=connected] 2024-08-31T22:19:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=13974], [object_number=65, tag_tid=8, entry_tid=13973], [object_number=64, tag_tid=8, entry_tid=13972], [object_number=67, tag_tid=8, entry_tid=13971]]], state=connected] 2024-08-31T22:19:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=66, tag_tid=8, entry_tid=13974' 2024-08-31T22:19:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=66, tag_tid=8, entry_tid=13974' 2024-08-31T22:19:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=13974' '!=' 'object_number=65, tag_tid=8, entry_tid=13869' 2024-08-31T22:19:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=13974' = 'object_number=65, tag_tid=8, entry_tid=13869' 2024-08-31T22:19:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=66, tag_tid=8, entry_tid=13974' 2024-08-31T22:19:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=13974' 2024-08-31T22:19:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=13974' 2024-08-31T22:19:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13974 2024-08-31T22:19:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 13974 2024-08-31T22:19:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:37.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:37.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:37.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:37.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:37.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=64, tag_tid=8, entry_tid=14076], [object_number=67, tag_tid=8, entry_tid=14075], [object_number=66, tag_tid=8, entry_tid=14074], [object_number=65, tag_tid=8, entry_tid=14073]]], state=connected] 2024-08-31T22:19:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=64, tag_tid=8, entry_tid=14076], [object_number=67, tag_tid=8, entry_tid=14075], [object_number=66, tag_tid=8, entry_tid=14074], [object_number=65, tag_tid=8, entry_tid=14073]]], state=connected] 2024-08-31T22:19:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=64, tag_tid=8, entry_tid=14076' 2024-08-31T22:19:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=64, tag_tid=8, entry_tid=14076' 2024-08-31T22:19:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=64, tag_tid=8, entry_tid=14076' '!=' 'object_number=66, tag_tid=8, entry_tid=13974' 2024-08-31T22:19:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=64, tag_tid=8, entry_tid=14076' = 'object_number=66, tag_tid=8, entry_tid=13974' 2024-08-31T22:19:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=64, tag_tid=8, entry_tid=14076' 2024-08-31T22:19:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=64, tag_tid=8, entry_tid=14076' 2024-08-31T22:19:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=64, tag_tid=8, entry_tid=14076' 2024-08-31T22:19:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14076 2024-08-31T22:19:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 14076 2024-08-31T22:19:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:38.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:38.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:38.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:38.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:38.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:38.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:38.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:38.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=14210], [object_number=65, tag_tid=8, entry_tid=14209], [object_number=64, tag_tid=8, entry_tid=14208], [object_number=67, tag_tid=8, entry_tid=14207]]], state=connected] 2024-08-31T22:19:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=14210], [object_number=65, tag_tid=8, entry_tid=14209], [object_number=64, tag_tid=8, entry_tid=14208], [object_number=67, tag_tid=8, entry_tid=14207]]], state=connected] 2024-08-31T22:19:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=66, tag_tid=8, entry_tid=14210' 2024-08-31T22:19:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=66, tag_tid=8, entry_tid=14210' 2024-08-31T22:19:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=14210' '!=' 'object_number=64, tag_tid=8, entry_tid=14076' 2024-08-31T22:19:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=14210' = 'object_number=64, tag_tid=8, entry_tid=14076' 2024-08-31T22:19:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=66, tag_tid=8, entry_tid=14210' 2024-08-31T22:19:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=14210' 2024-08-31T22:19:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=14210' 2024-08-31T22:19:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14210 2024-08-31T22:19:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 14210 2024-08-31T22:19:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:38.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:39.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:39.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:39.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:39.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:39.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:39.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:39.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:39.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:39.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:39.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=65, tag_tid=8, entry_tid=14301], [object_number=64, tag_tid=8, entry_tid=14300], [object_number=67, tag_tid=8, entry_tid=14299], [object_number=66, tag_tid=8, entry_tid=14298]]], state=connected] 2024-08-31T22:19:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=65, tag_tid=8, entry_tid=14301], [object_number=64, tag_tid=8, entry_tid=14300], [object_number=67, tag_tid=8, entry_tid=14299], [object_number=66, tag_tid=8, entry_tid=14298]]], state=connected] 2024-08-31T22:19:39.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=65, tag_tid=8, entry_tid=14301' 2024-08-31T22:19:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=65, tag_tid=8, entry_tid=14301' 2024-08-31T22:19:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=65, tag_tid=8, entry_tid=14301' '!=' 'object_number=66, tag_tid=8, entry_tid=14210' 2024-08-31T22:19:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=65, tag_tid=8, entry_tid=14301' = 'object_number=66, tag_tid=8, entry_tid=14210' 2024-08-31T22:19:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=65, tag_tid=8, entry_tid=14301' 2024-08-31T22:19:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=65, tag_tid=8, entry_tid=14301' 2024-08-31T22:19:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=65, tag_tid=8, entry_tid=14301' 2024-08-31T22:19:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14301 2024-08-31T22:19:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 14301 2024-08-31T22:19:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:39.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:39.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:39.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:39.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:39.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:39.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:39.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:39.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=67, tag_tid=8, entry_tid=14395], [object_number=66, tag_tid=8, entry_tid=14394], [object_number=65, tag_tid=8, entry_tid=14393], [object_number=64, tag_tid=8, entry_tid=14392]]], state=connected] 2024-08-31T22:19:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=67, tag_tid=8, entry_tid=14395], [object_number=66, tag_tid=8, entry_tid=14394], [object_number=65, tag_tid=8, entry_tid=14393], [object_number=64, tag_tid=8, entry_tid=14392]]], state=connected] 2024-08-31T22:19:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=67, tag_tid=8, entry_tid=14395' 2024-08-31T22:19:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=67, tag_tid=8, entry_tid=14395' 2024-08-31T22:19:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=67, tag_tid=8, entry_tid=14395' '!=' 'object_number=65, tag_tid=8, entry_tid=14301' 2024-08-31T22:19:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=67, tag_tid=8, entry_tid=14395' = 'object_number=65, tag_tid=8, entry_tid=14301' 2024-08-31T22:19:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=67, tag_tid=8, entry_tid=14395' 2024-08-31T22:19:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=67, tag_tid=8, entry_tid=14395' 2024-08-31T22:19:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=67, tag_tid=8, entry_tid=14395' 2024-08-31T22:19:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14395 2024-08-31T22:19:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 14395 2024-08-31T22:19:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=65, tag_tid=8, entry_tid=14501], [object_number=64, tag_tid=8, entry_tid=14500], [object_number=67, tag_tid=8, entry_tid=14499], [object_number=66, tag_tid=8, entry_tid=14498]]], state=connected] 2024-08-31T22:19:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=65, tag_tid=8, entry_tid=14501], [object_number=64, tag_tid=8, entry_tid=14500], [object_number=67, tag_tid=8, entry_tid=14499], [object_number=66, tag_tid=8, entry_tid=14498]]], state=connected] 2024-08-31T22:19:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=65, tag_tid=8, entry_tid=14501' 2024-08-31T22:19:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=65, tag_tid=8, entry_tid=14501' 2024-08-31T22:19:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=65, tag_tid=8, entry_tid=14501' '!=' 'object_number=67, tag_tid=8, entry_tid=14395' 2024-08-31T22:19:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=65, tag_tid=8, entry_tid=14501' = 'object_number=67, tag_tid=8, entry_tid=14395' 2024-08-31T22:19:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=65, tag_tid=8, entry_tid=14501' 2024-08-31T22:19:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=65, tag_tid=8, entry_tid=14501' 2024-08-31T22:19:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=65, tag_tid=8, entry_tid=14501' 2024-08-31T22:19:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14501 2024-08-31T22:19:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 14501 2024-08-31T22:19:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:40.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:40.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:40.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:40.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:40.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:40.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=14610], [object_number=65, tag_tid=8, entry_tid=14609], [object_number=64, tag_tid=8, entry_tid=14608], [object_number=67, tag_tid=8, entry_tid=14607]]], state=connected] 2024-08-31T22:19:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=14610], [object_number=65, tag_tid=8, entry_tid=14609], [object_number=64, tag_tid=8, entry_tid=14608], [object_number=67, tag_tid=8, entry_tid=14607]]], state=connected] 2024-08-31T22:19:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=66, tag_tid=8, entry_tid=14610' 2024-08-31T22:19:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=66, tag_tid=8, entry_tid=14610' 2024-08-31T22:19:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=14610' '!=' 'object_number=65, tag_tid=8, entry_tid=14501' 2024-08-31T22:19:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=14610' = 'object_number=65, tag_tid=8, entry_tid=14501' 2024-08-31T22:19:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=66, tag_tid=8, entry_tid=14610' 2024-08-31T22:19:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=14610' 2024-08-31T22:19:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=14610' 2024-08-31T22:19:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14610 2024-08-31T22:19:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 14610 2024-08-31T22:19:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=65, tag_tid=8, entry_tid=14713], [object_number=64, tag_tid=8, entry_tid=14712], [object_number=67, tag_tid=8, entry_tid=14711], [object_number=66, tag_tid=8, entry_tid=14710]]], state=connected] 2024-08-31T22:19:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=65, tag_tid=8, entry_tid=14713], [object_number=64, tag_tid=8, entry_tid=14712], [object_number=67, tag_tid=8, entry_tid=14711], [object_number=66, tag_tid=8, entry_tid=14710]]], state=connected] 2024-08-31T22:19:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=65, tag_tid=8, entry_tid=14713' 2024-08-31T22:19:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=65, tag_tid=8, entry_tid=14713' 2024-08-31T22:19:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=65, tag_tid=8, entry_tid=14713' '!=' 'object_number=66, tag_tid=8, entry_tid=14610' 2024-08-31T22:19:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=65, tag_tid=8, entry_tid=14713' = 'object_number=66, tag_tid=8, entry_tid=14610' 2024-08-31T22:19:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=65, tag_tid=8, entry_tid=14713' 2024-08-31T22:19:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=65, tag_tid=8, entry_tid=14713' 2024-08-31T22:19:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=65, tag_tid=8, entry_tid=14713' 2024-08-31T22:19:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14713 2024-08-31T22:19:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 14713 2024-08-31T22:19:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=67, tag_tid=8, entry_tid=14819], [object_number=66, tag_tid=8, entry_tid=14818], [object_number=65, tag_tid=8, entry_tid=14817], [object_number=64, tag_tid=8, entry_tid=14816]]], state=connected] 2024-08-31T22:19:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=67, tag_tid=8, entry_tid=14819], [object_number=66, tag_tid=8, entry_tid=14818], [object_number=65, tag_tid=8, entry_tid=14817], [object_number=64, tag_tid=8, entry_tid=14816]]], state=connected] 2024-08-31T22:19:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=67, tag_tid=8, entry_tid=14819' 2024-08-31T22:19:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=67, tag_tid=8, entry_tid=14819' 2024-08-31T22:19:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=67, tag_tid=8, entry_tid=14819' '!=' 'object_number=65, tag_tid=8, entry_tid=14713' 2024-08-31T22:19:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=67, tag_tid=8, entry_tid=14819' = 'object_number=65, tag_tid=8, entry_tid=14713' 2024-08-31T22:19:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=67, tag_tid=8, entry_tid=14819' 2024-08-31T22:19:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=67, tag_tid=8, entry_tid=14819' 2024-08-31T22:19:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:41.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=67, tag_tid=8, entry_tid=14819' 2024-08-31T22:19:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14819 2024-08-31T22:19:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 14819 2024-08-31T22:19:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:41.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:41.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:41.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:41.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:41.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:41.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:41.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:41.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=65, tag_tid=8, entry_tid=14921], [object_number=64, tag_tid=8, entry_tid=14920], [object_number=67, tag_tid=8, entry_tid=14919], [object_number=66, tag_tid=8, entry_tid=14918]]], state=connected] 2024-08-31T22:19:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=65, tag_tid=8, entry_tid=14921], [object_number=64, tag_tid=8, entry_tid=14920], [object_number=67, tag_tid=8, entry_tid=14919], [object_number=66, tag_tid=8, entry_tid=14918]]], state=connected] 2024-08-31T22:19:42.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=65, tag_tid=8, entry_tid=14921' 2024-08-31T22:19:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=65, tag_tid=8, entry_tid=14921' 2024-08-31T22:19:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=65, tag_tid=8, entry_tid=14921' '!=' 'object_number=67, tag_tid=8, entry_tid=14819' 2024-08-31T22:19:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=65, tag_tid=8, entry_tid=14921' = 'object_number=67, tag_tid=8, entry_tid=14819' 2024-08-31T22:19:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=65, tag_tid=8, entry_tid=14921' 2024-08-31T22:19:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:42.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:42.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:42.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:42.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=65, tag_tid=8, entry_tid=14921' 2024-08-31T22:19:42.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:42.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=65, tag_tid=8, entry_tid=14921' 2024-08-31T22:19:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14921 2024-08-31T22:19:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 14921 2024-08-31T22:19:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=15018], [object_number=65, tag_tid=8, entry_tid=15017], [object_number=64, tag_tid=8, entry_tid=15016], [object_number=67, tag_tid=8, entry_tid=15015]]], state=connected] 2024-08-31T22:19:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=15018], [object_number=65, tag_tid=8, entry_tid=15017], [object_number=64, tag_tid=8, entry_tid=15016], [object_number=67, tag_tid=8, entry_tid=15015]]], state=connected] 2024-08-31T22:19:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=66, tag_tid=8, entry_tid=15018' 2024-08-31T22:19:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=66, tag_tid=8, entry_tid=15018' 2024-08-31T22:19:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=15018' '!=' 'object_number=65, tag_tid=8, entry_tid=14921' 2024-08-31T22:19:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=15018' = 'object_number=65, tag_tid=8, entry_tid=14921' 2024-08-31T22:19:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=66, tag_tid=8, entry_tid=15018' 2024-08-31T22:19:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=15018' 2024-08-31T22:19:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:42.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:42.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:42.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=15018' 2024-08-31T22:19:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15018 2024-08-31T22:19:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 15018 2024-08-31T22:19:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=15118], [object_number=65, tag_tid=8, entry_tid=15117], [object_number=64, tag_tid=8, entry_tid=15116], [object_number=67, tag_tid=8, entry_tid=15115]]], state=connected] 2024-08-31T22:19:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=15118], [object_number=65, tag_tid=8, entry_tid=15117], [object_number=64, tag_tid=8, entry_tid=15116], [object_number=67, tag_tid=8, entry_tid=15115]]], state=connected] 2024-08-31T22:19:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=66, tag_tid=8, entry_tid=15118' 2024-08-31T22:19:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=66, tag_tid=8, entry_tid=15118' 2024-08-31T22:19:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=15118' '!=' 'object_number=66, tag_tid=8, entry_tid=15018' 2024-08-31T22:19:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=15118' = 'object_number=66, tag_tid=8, entry_tid=15018' 2024-08-31T22:19:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=66, tag_tid=8, entry_tid=15118' 2024-08-31T22:19:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=15118' 2024-08-31T22:19:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=15118' 2024-08-31T22:19:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15118 2024-08-31T22:19:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 15118 2024-08-31T22:19:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:43.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=67, tag_tid=8, entry_tid=15215], [object_number=66, tag_tid=8, entry_tid=15214], [object_number=65, tag_tid=8, entry_tid=15213], [object_number=64, tag_tid=8, entry_tid=15212]]], state=connected] 2024-08-31T22:19:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=67, tag_tid=8, entry_tid=15215], [object_number=66, tag_tid=8, entry_tid=15214], [object_number=65, tag_tid=8, entry_tid=15213], [object_number=64, tag_tid=8, entry_tid=15212]]], state=connected] 2024-08-31T22:19:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=67, tag_tid=8, entry_tid=15215' 2024-08-31T22:19:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=67, tag_tid=8, entry_tid=15215' 2024-08-31T22:19:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=67, tag_tid=8, entry_tid=15215' '!=' 'object_number=66, tag_tid=8, entry_tid=15118' 2024-08-31T22:19:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=67, tag_tid=8, entry_tid=15215' = 'object_number=66, tag_tid=8, entry_tid=15118' 2024-08-31T22:19:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=67, tag_tid=8, entry_tid=15215' 2024-08-31T22:19:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=67, tag_tid=8, entry_tid=15215' 2024-08-31T22:19:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=67, tag_tid=8, entry_tid=15215' 2024-08-31T22:19:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15215 2024-08-31T22:19:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 15215 2024-08-31T22:19:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:43.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:43.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:43.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:43.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:43.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:43.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:43.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:43.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:43.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:43.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:43.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=64, tag_tid=8, entry_tid=15324], [object_number=67, tag_tid=8, entry_tid=15323], [object_number=66, tag_tid=8, entry_tid=15322], [object_number=65, tag_tid=8, entry_tid=15321]]], state=connected] 2024-08-31T22:19:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:44.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:44.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:44.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:44.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:44.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=64, tag_tid=8, entry_tid=15324], [object_number=67, tag_tid=8, entry_tid=15323], [object_number=66, tag_tid=8, entry_tid=15322], [object_number=65, tag_tid=8, entry_tid=15321]]], state=connected] 2024-08-31T22:19:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=64, tag_tid=8, entry_tid=15324' 2024-08-31T22:19:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=64, tag_tid=8, entry_tid=15324' 2024-08-31T22:19:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=64, tag_tid=8, entry_tid=15324' '!=' 'object_number=67, tag_tid=8, entry_tid=15215' 2024-08-31T22:19:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=64, tag_tid=8, entry_tid=15324' = 'object_number=67, tag_tid=8, entry_tid=15215' 2024-08-31T22:19:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=64, tag_tid=8, entry_tid=15324' 2024-08-31T22:19:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=64, tag_tid=8, entry_tid=15324' 2024-08-31T22:19:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=64, tag_tid=8, entry_tid=15324' 2024-08-31T22:19:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15324 2024-08-31T22:19:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 15324 2024-08-31T22:19:44.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:44.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:44.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=15418], [object_number=65, tag_tid=8, entry_tid=15417], [object_number=64, tag_tid=8, entry_tid=15416], [object_number=67, tag_tid=8, entry_tid=15415]]], state=connected] 2024-08-31T22:19:44.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=15418], [object_number=65, tag_tid=8, entry_tid=15417], [object_number=64, tag_tid=8, entry_tid=15416], [object_number=67, tag_tid=8, entry_tid=15415]]], state=connected] 2024-08-31T22:19:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=66, tag_tid=8, entry_tid=15418' 2024-08-31T22:19:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=66, tag_tid=8, entry_tid=15418' 2024-08-31T22:19:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=15418' '!=' 'object_number=64, tag_tid=8, entry_tid=15324' 2024-08-31T22:19:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=15418' = 'object_number=64, tag_tid=8, entry_tid=15324' 2024-08-31T22:19:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=66, tag_tid=8, entry_tid=15418' 2024-08-31T22:19:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=15418' 2024-08-31T22:19:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=15418' 2024-08-31T22:19:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15418 2024-08-31T22:19:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 15418 2024-08-31T22:19:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:45.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:45.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:45.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:45.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:45.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:45.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:45.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:45.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:45.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:45.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:45.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:45.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:45.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:45.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:45.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=67, tag_tid=8, entry_tid=15547], [object_number=66, tag_tid=8, entry_tid=15546], [object_number=65, tag_tid=8, entry_tid=15545], [object_number=64, tag_tid=8, entry_tid=15544]]], state=connected] 2024-08-31T22:19:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=67, tag_tid=8, entry_tid=15547], [object_number=66, tag_tid=8, entry_tid=15546], [object_number=65, tag_tid=8, entry_tid=15545], [object_number=64, tag_tid=8, entry_tid=15544]]], state=connected] 2024-08-31T22:19:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=67, tag_tid=8, entry_tid=15547' 2024-08-31T22:19:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=67, tag_tid=8, entry_tid=15547' 2024-08-31T22:19:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=67, tag_tid=8, entry_tid=15547' '!=' 'object_number=66, tag_tid=8, entry_tid=15418' 2024-08-31T22:19:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=67, tag_tid=8, entry_tid=15547' = 'object_number=66, tag_tid=8, entry_tid=15418' 2024-08-31T22:19:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=67, tag_tid=8, entry_tid=15547' 2024-08-31T22:19:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=67, tag_tid=8, entry_tid=15547' 2024-08-31T22:19:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=67, tag_tid=8, entry_tid=15547' 2024-08-31T22:19:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15547 2024-08-31T22:19:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 15547 2024-08-31T22:19:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:45.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:45.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:45.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:45.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:45.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:45.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:45.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:45.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:45.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=67, tag_tid=8, entry_tid=15643], [object_number=66, tag_tid=8, entry_tid=15642], [object_number=65, tag_tid=8, entry_tid=15641], [object_number=64, tag_tid=8, entry_tid=15640]]], state=connected] 2024-08-31T22:19:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=67, tag_tid=8, entry_tid=15643], [object_number=66, tag_tid=8, entry_tid=15642], [object_number=65, tag_tid=8, entry_tid=15641], [object_number=64, tag_tid=8, entry_tid=15640]]], state=connected] 2024-08-31T22:19:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=67, tag_tid=8, entry_tid=15643' 2024-08-31T22:19:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=67, tag_tid=8, entry_tid=15643' 2024-08-31T22:19:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=67, tag_tid=8, entry_tid=15643' '!=' 'object_number=67, tag_tid=8, entry_tid=15547' 2024-08-31T22:19:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=67, tag_tid=8, entry_tid=15643' = 'object_number=67, tag_tid=8, entry_tid=15547' 2024-08-31T22:19:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=67, tag_tid=8, entry_tid=15643' 2024-08-31T22:19:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=67, tag_tid=8, entry_tid=15643' 2024-08-31T22:19:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:45.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=67, tag_tid=8, entry_tid=15643' 2024-08-31T22:19:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15643 2024-08-31T22:19:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 15643 2024-08-31T22:19:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:46.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:46.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:46.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=15754], [object_number=65, tag_tid=8, entry_tid=15753], [object_number=64, tag_tid=8, entry_tid=15752], [object_number=67, tag_tid=8, entry_tid=15751]]], state=connected] 2024-08-31T22:19:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=15754], [object_number=65, tag_tid=8, entry_tid=15753], [object_number=64, tag_tid=8, entry_tid=15752], [object_number=67, tag_tid=8, entry_tid=15751]]], state=connected] 2024-08-31T22:19:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=66, tag_tid=8, entry_tid=15754' 2024-08-31T22:19:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=66, tag_tid=8, entry_tid=15754' 2024-08-31T22:19:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=15754' '!=' 'object_number=67, tag_tid=8, entry_tid=15643' 2024-08-31T22:19:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=15754' = 'object_number=67, tag_tid=8, entry_tid=15643' 2024-08-31T22:19:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=66, tag_tid=8, entry_tid=15754' 2024-08-31T22:19:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=15754' 2024-08-31T22:19:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=15754' 2024-08-31T22:19:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15754 2024-08-31T22:19:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 15754 2024-08-31T22:19:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=67, tag_tid=8, entry_tid=15855], [object_number=66, tag_tid=8, entry_tid=15854], [object_number=65, tag_tid=8, entry_tid=15853], [object_number=64, tag_tid=8, entry_tid=15852]]], state=connected] 2024-08-31T22:19:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=67, tag_tid=8, entry_tid=15855], [object_number=66, tag_tid=8, entry_tid=15854], [object_number=65, tag_tid=8, entry_tid=15853], [object_number=64, tag_tid=8, entry_tid=15852]]], state=connected] 2024-08-31T22:19:46.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:46.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=67, tag_tid=8, entry_tid=15855' 2024-08-31T22:19:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=67, tag_tid=8, entry_tid=15855' 2024-08-31T22:19:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=67, tag_tid=8, entry_tid=15855' '!=' 'object_number=66, tag_tid=8, entry_tid=15754' 2024-08-31T22:19:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=67, tag_tid=8, entry_tid=15855' = 'object_number=66, tag_tid=8, entry_tid=15754' 2024-08-31T22:19:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=67, tag_tid=8, entry_tid=15855' 2024-08-31T22:19:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=67, tag_tid=8, entry_tid=15855' 2024-08-31T22:19:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=67, tag_tid=8, entry_tid=15855' 2024-08-31T22:19:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15855 2024-08-31T22:19:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 15855 2024-08-31T22:19:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=15950], [object_number=65, tag_tid=8, entry_tid=15949], [object_number=64, tag_tid=8, entry_tid=15948], [object_number=67, tag_tid=8, entry_tid=15947]]], state=connected] 2024-08-31T22:19:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=15950], [object_number=65, tag_tid=8, entry_tid=15949], [object_number=64, tag_tid=8, entry_tid=15948], [object_number=67, tag_tid=8, entry_tid=15947]]], state=connected] 2024-08-31T22:19:47.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=66, tag_tid=8, entry_tid=15950' 2024-08-31T22:19:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=66, tag_tid=8, entry_tid=15950' 2024-08-31T22:19:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=15950' '!=' 'object_number=67, tag_tid=8, entry_tid=15855' 2024-08-31T22:19:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=15950' = 'object_number=67, tag_tid=8, entry_tid=15855' 2024-08-31T22:19:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=66, tag_tid=8, entry_tid=15950' 2024-08-31T22:19:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=15950' 2024-08-31T22:19:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=15950' 2024-08-31T22:19:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15950 2024-08-31T22:19:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 15950 2024-08-31T22:19:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:47.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:47.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:47.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:47.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=64, tag_tid=8, entry_tid=16044], [object_number=67, tag_tid=8, entry_tid=16043], [object_number=66, tag_tid=8, entry_tid=16042], [object_number=65, tag_tid=8, entry_tid=16041]]], state=connected] 2024-08-31T22:19:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=64, tag_tid=8, entry_tid=16044], [object_number=67, tag_tid=8, entry_tid=16043], [object_number=66, tag_tid=8, entry_tid=16042], [object_number=65, tag_tid=8, entry_tid=16041]]], state=connected] 2024-08-31T22:19:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=64, tag_tid=8, entry_tid=16044' 2024-08-31T22:19:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=64, tag_tid=8, entry_tid=16044' 2024-08-31T22:19:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=64, tag_tid=8, entry_tid=16044' '!=' 'object_number=66, tag_tid=8, entry_tid=15950' 2024-08-31T22:19:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=64, tag_tid=8, entry_tid=16044' = 'object_number=66, tag_tid=8, entry_tid=15950' 2024-08-31T22:19:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=64, tag_tid=8, entry_tid=16044' 2024-08-31T22:19:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=64, tag_tid=8, entry_tid=16044' 2024-08-31T22:19:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=64, tag_tid=8, entry_tid=16044' 2024-08-31T22:19:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16044 2024-08-31T22:19:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 16044 2024-08-31T22:19:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=16134], [object_number=65, tag_tid=8, entry_tid=16133], [object_number=64, tag_tid=8, entry_tid=16132], [object_number=67, tag_tid=8, entry_tid=16131]]], state=connected] 2024-08-31T22:19:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:48.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=16134], [object_number=65, tag_tid=8, entry_tid=16133], [object_number=64, tag_tid=8, entry_tid=16132], [object_number=67, tag_tid=8, entry_tid=16131]]], state=connected] 2024-08-31T22:19:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=66, tag_tid=8, entry_tid=16134' 2024-08-31T22:19:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=66, tag_tid=8, entry_tid=16134' 2024-08-31T22:19:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=16134' '!=' 'object_number=64, tag_tid=8, entry_tid=16044' 2024-08-31T22:19:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=16134' = 'object_number=64, tag_tid=8, entry_tid=16044' 2024-08-31T22:19:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=66, tag_tid=8, entry_tid=16134' 2024-08-31T22:19:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=16134' 2024-08-31T22:19:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=16134' 2024-08-31T22:19:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16134 2024-08-31T22:19:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 16134 2024-08-31T22:19:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=16234], [object_number=65, tag_tid=8, entry_tid=16233], [object_number=64, tag_tid=8, entry_tid=16232], [object_number=67, tag_tid=8, entry_tid=16231]]], state=connected] 2024-08-31T22:19:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=16234], [object_number=65, tag_tid=8, entry_tid=16233], [object_number=64, tag_tid=8, entry_tid=16232], [object_number=67, tag_tid=8, entry_tid=16231]]], state=connected] 2024-08-31T22:19:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=66, tag_tid=8, entry_tid=16234' 2024-08-31T22:19:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=66, tag_tid=8, entry_tid=16234' 2024-08-31T22:19:48.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=16234' '!=' 'object_number=66, tag_tid=8, entry_tid=16134' 2024-08-31T22:19:48.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:48.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=16234' = 'object_number=66, tag_tid=8, entry_tid=16134' 2024-08-31T22:19:48.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=66, tag_tid=8, entry_tid=16234' 2024-08-31T22:19:48.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:48.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:48.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:48.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=16234' 2024-08-31T22:19:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=16234' 2024-08-31T22:19:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16234 2024-08-31T22:19:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 16234 2024-08-31T22:19:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=64, tag_tid=8, entry_tid=16324], [object_number=67, tag_tid=8, entry_tid=16323], [object_number=66, tag_tid=8, entry_tid=16322], [object_number=65, tag_tid=8, entry_tid=16321]]], state=connected] 2024-08-31T22:19:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=64, tag_tid=8, entry_tid=16324], [object_number=67, tag_tid=8, entry_tid=16323], [object_number=66, tag_tid=8, entry_tid=16322], [object_number=65, tag_tid=8, entry_tid=16321]]], state=connected] 2024-08-31T22:19:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=64, tag_tid=8, entry_tid=16324' 2024-08-31T22:19:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=64, tag_tid=8, entry_tid=16324' 2024-08-31T22:19:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=64, tag_tid=8, entry_tid=16324' '!=' 'object_number=66, tag_tid=8, entry_tid=16234' 2024-08-31T22:19:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=64, tag_tid=8, entry_tid=16324' = 'object_number=66, tag_tid=8, entry_tid=16234' 2024-08-31T22:19:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=64, tag_tid=8, entry_tid=16324' 2024-08-31T22:19:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=64, tag_tid=8, entry_tid=16324' 2024-08-31T22:19:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=64, tag_tid=8, entry_tid=16324' 2024-08-31T22:19:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16324 2024-08-31T22:19:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 16324 2024-08-31T22:19:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:49.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:49.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=67, tag_tid=8, entry_tid=16443], [object_number=66, tag_tid=8, entry_tid=16442], [object_number=65, tag_tid=8, entry_tid=16441], [object_number=64, tag_tid=8, entry_tid=16440]]], state=connected] 2024-08-31T22:19:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:49.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=67, tag_tid=8, entry_tid=16443], [object_number=66, tag_tid=8, entry_tid=16442], [object_number=65, tag_tid=8, entry_tid=16441], [object_number=64, tag_tid=8, entry_tid=16440]]], state=connected] 2024-08-31T22:19:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=67, tag_tid=8, entry_tid=16443' 2024-08-31T22:19:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=67, tag_tid=8, entry_tid=16443' 2024-08-31T22:19:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=67, tag_tid=8, entry_tid=16443' '!=' 'object_number=64, tag_tid=8, entry_tid=16324' 2024-08-31T22:19:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=67, tag_tid=8, entry_tid=16443' = 'object_number=64, tag_tid=8, entry_tid=16324' 2024-08-31T22:19:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=67, tag_tid=8, entry_tid=16443' 2024-08-31T22:19:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=67, tag_tid=8, entry_tid=16443' 2024-08-31T22:19:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=67, tag_tid=8, entry_tid=16443' 2024-08-31T22:19:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16443 2024-08-31T22:19:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 16443 2024-08-31T22:19:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=64, tag_tid=8, entry_tid=16532], [object_number=67, tag_tid=8, entry_tid=16531], [object_number=66, tag_tid=8, entry_tid=16530], [object_number=65, tag_tid=8, entry_tid=16529]]], state=connected] 2024-08-31T22:19:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=64, tag_tid=8, entry_tid=16532], [object_number=67, tag_tid=8, entry_tid=16531], [object_number=66, tag_tid=8, entry_tid=16530], [object_number=65, tag_tid=8, entry_tid=16529]]], state=connected] 2024-08-31T22:19:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=64, tag_tid=8, entry_tid=16532' 2024-08-31T22:19:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=64, tag_tid=8, entry_tid=16532' 2024-08-31T22:19:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=64, tag_tid=8, entry_tid=16532' '!=' 'object_number=67, tag_tid=8, entry_tid=16443' 2024-08-31T22:19:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=64, tag_tid=8, entry_tid=16532' = 'object_number=67, tag_tid=8, entry_tid=16443' 2024-08-31T22:19:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=64, tag_tid=8, entry_tid=16532' 2024-08-31T22:19:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=64, tag_tid=8, entry_tid=16532' 2024-08-31T22:19:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=64, tag_tid=8, entry_tid=16532' 2024-08-31T22:19:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16532 2024-08-31T22:19:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 16532 2024-08-31T22:19:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=67, tag_tid=8, entry_tid=16619], [object_number=66, tag_tid=8, entry_tid=16618], [object_number=65, tag_tid=8, entry_tid=16617], [object_number=64, tag_tid=8, entry_tid=16616]]], state=connected] 2024-08-31T22:19:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=67, tag_tid=8, entry_tid=16619], [object_number=66, tag_tid=8, entry_tid=16618], [object_number=65, tag_tid=8, entry_tid=16617], [object_number=64, tag_tid=8, entry_tid=16616]]], state=connected] 2024-08-31T22:19:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=67, tag_tid=8, entry_tid=16619' 2024-08-31T22:19:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=67, tag_tid=8, entry_tid=16619' 2024-08-31T22:19:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=67, tag_tid=8, entry_tid=16619' '!=' 'object_number=64, tag_tid=8, entry_tid=16532' 2024-08-31T22:19:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=67, tag_tid=8, entry_tid=16619' = 'object_number=64, tag_tid=8, entry_tid=16532' 2024-08-31T22:19:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=67, tag_tid=8, entry_tid=16619' 2024-08-31T22:19:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=67, tag_tid=8, entry_tid=16619' 2024-08-31T22:19:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=67, tag_tid=8, entry_tid=16619' 2024-08-31T22:19:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16619 2024-08-31T22:19:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 16619 2024-08-31T22:19:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=65, tag_tid=8, entry_tid=16705], [object_number=64, tag_tid=8, entry_tid=16704], [object_number=67, tag_tid=8, entry_tid=16703], [object_number=66, tag_tid=8, entry_tid=16702]]], state=connected] 2024-08-31T22:19:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=65, tag_tid=8, entry_tid=16705], [object_number=64, tag_tid=8, entry_tid=16704], [object_number=67, tag_tid=8, entry_tid=16703], [object_number=66, tag_tid=8, entry_tid=16702]]], state=connected] 2024-08-31T22:19:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=65, tag_tid=8, entry_tid=16705' 2024-08-31T22:19:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=65, tag_tid=8, entry_tid=16705' 2024-08-31T22:19:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=65, tag_tid=8, entry_tid=16705' '!=' 'object_number=67, tag_tid=8, entry_tid=16619' 2024-08-31T22:19:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=65, tag_tid=8, entry_tid=16705' = 'object_number=67, tag_tid=8, entry_tid=16619' 2024-08-31T22:19:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=65, tag_tid=8, entry_tid=16705' 2024-08-31T22:19:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=65, tag_tid=8, entry_tid=16705' 2024-08-31T22:19:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=65, tag_tid=8, entry_tid=16705' 2024-08-31T22:19:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16705 2024-08-31T22:19:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 16705 2024-08-31T22:19:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=16802], [object_number=65, tag_tid=8, entry_tid=16801], [object_number=64, tag_tid=8, entry_tid=16800], [object_number=67, tag_tid=8, entry_tid=16799]]], state=connected] 2024-08-31T22:19:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=66, tag_tid=8, entry_tid=16802], [object_number=65, tag_tid=8, entry_tid=16801], [object_number=64, tag_tid=8, entry_tid=16800], [object_number=67, tag_tid=8, entry_tid=16799]]], state=connected] 2024-08-31T22:19:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=66, tag_tid=8, entry_tid=16802' 2024-08-31T22:19:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=66, tag_tid=8, entry_tid=16802' 2024-08-31T22:19:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=16802' '!=' 'object_number=65, tag_tid=8, entry_tid=16705' 2024-08-31T22:19:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=66, tag_tid=8, entry_tid=16802' = 'object_number=65, tag_tid=8, entry_tid=16705' 2024-08-31T22:19:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=66, tag_tid=8, entry_tid=16802' 2024-08-31T22:19:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=16802' 2024-08-31T22:19:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=66, tag_tid=8, entry_tid=16802' 2024-08-31T22:19:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16802 2024-08-31T22:19:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 16802 2024-08-31T22:19:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:52.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:52.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:52.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:52.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=16883], [object_number=66, tag_tid=8, entry_tid=16882], [object_number=65, tag_tid=8, entry_tid=16881], [object_number=64, tag_tid=8, entry_tid=16880]]], state=connected] 2024-08-31T22:19:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 16 2024-08-31T22:19:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=16883], [object_number=66, tag_tid=8, entry_tid=16882], [object_number=65, tag_tid=8, entry_tid=16881], [object_number=64, tag_tid=8, entry_tid=16880]]], state=connected] 2024-08-31T22:19:52.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:52.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=71, tag_tid=8, entry_tid=16883' 2024-08-31T22:19:52.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=71, tag_tid=8, entry_tid=16883' 2024-08-31T22:19:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=16883' '!=' 'object_number=66, tag_tid=8, entry_tid=16802' 2024-08-31T22:19:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=16883' = 'object_number=66, tag_tid=8, entry_tid=16802' 2024-08-31T22:19:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=71, tag_tid=8, entry_tid=16883' 2024-08-31T22:19:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=16883' 2024-08-31T22:19:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:52.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:52.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=16883' 2024-08-31T22:19:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16883 2024-08-31T22:19:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 16883 2024-08-31T22:19:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=16931], [object_number=70, tag_tid=8, entry_tid=16930], [object_number=69, tag_tid=8, entry_tid=16929], [object_number=68, tag_tid=8, entry_tid=16928]]], state=connected] 2024-08-31T22:19:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:53.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:53.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=16931], [object_number=70, tag_tid=8, entry_tid=16930], [object_number=69, tag_tid=8, entry_tid=16929], [object_number=68, tag_tid=8, entry_tid=16928]]], state=connected] 2024-08-31T22:19:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=71, tag_tid=8, entry_tid=16931' 2024-08-31T22:19:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=71, tag_tid=8, entry_tid=16931' 2024-08-31T22:19:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=16931' '!=' 'object_number=71, tag_tid=8, entry_tid=16883' 2024-08-31T22:19:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=16931' = 'object_number=71, tag_tid=8, entry_tid=16883' 2024-08-31T22:19:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=71, tag_tid=8, entry_tid=16931' 2024-08-31T22:19:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=16931' 2024-08-31T22:19:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:53.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:53.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=16931' 2024-08-31T22:19:53.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:53.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:53.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16931 2024-08-31T22:19:53.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 16931 2024-08-31T22:19:53.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:53.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:53.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:53.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:53.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:53.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:53.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:53.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:53.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:53.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:53.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:53.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:53.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:53.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:53.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:53.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:53.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:53.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=17031], [object_number=70, tag_tid=8, entry_tid=17030], [object_number=69, tag_tid=8, entry_tid=17029], [object_number=68, tag_tid=8, entry_tid=17028]]], state=connected] 2024-08-31T22:19:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:53.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=17031], [object_number=70, tag_tid=8, entry_tid=17030], [object_number=69, tag_tid=8, entry_tid=17029], [object_number=68, tag_tid=8, entry_tid=17028]]], state=connected] 2024-08-31T22:19:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:53.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=71, tag_tid=8, entry_tid=17031' 2024-08-31T22:19:53.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=71, tag_tid=8, entry_tid=17031' 2024-08-31T22:19:53.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=17031' '!=' 'object_number=71, tag_tid=8, entry_tid=16931' 2024-08-31T22:19:53.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:53.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=17031' = 'object_number=71, tag_tid=8, entry_tid=16931' 2024-08-31T22:19:53.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=71, tag_tid=8, entry_tid=17031' 2024-08-31T22:19:53.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:53.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:53.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=17031' 2024-08-31T22:19:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=17031' 2024-08-31T22:19:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17031 2024-08-31T22:19:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 17031 2024-08-31T22:19:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=17131], [object_number=70, tag_tid=8, entry_tid=17130], [object_number=69, tag_tid=8, entry_tid=17129], [object_number=68, tag_tid=8, entry_tid=17128]]], state=connected] 2024-08-31T22:19:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:53.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=17131], [object_number=70, tag_tid=8, entry_tid=17130], [object_number=69, tag_tid=8, entry_tid=17129], [object_number=68, tag_tid=8, entry_tid=17128]]], state=connected] 2024-08-31T22:19:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=71, tag_tid=8, entry_tid=17131' 2024-08-31T22:19:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=71, tag_tid=8, entry_tid=17131' 2024-08-31T22:19:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=17131' '!=' 'object_number=71, tag_tid=8, entry_tid=17031' 2024-08-31T22:19:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=17131' = 'object_number=71, tag_tid=8, entry_tid=17031' 2024-08-31T22:19:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=71, tag_tid=8, entry_tid=17131' 2024-08-31T22:19:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=17131' 2024-08-31T22:19:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=17131' 2024-08-31T22:19:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:54.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17131 2024-08-31T22:19:54.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 17131 2024-08-31T22:19:54.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:54.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:54.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=68, tag_tid=8, entry_tid=17220], [object_number=71, tag_tid=8, entry_tid=17219], [object_number=70, tag_tid=8, entry_tid=17218], [object_number=69, tag_tid=8, entry_tid=17217]]], state=connected] 2024-08-31T22:19:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=68, tag_tid=8, entry_tid=17220], [object_number=71, tag_tid=8, entry_tid=17219], [object_number=70, tag_tid=8, entry_tid=17218], [object_number=69, tag_tid=8, entry_tid=17217]]], state=connected] 2024-08-31T22:19:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=68, tag_tid=8, entry_tid=17220' 2024-08-31T22:19:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=68, tag_tid=8, entry_tid=17220' 2024-08-31T22:19:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=68, tag_tid=8, entry_tid=17220' '!=' 'object_number=71, tag_tid=8, entry_tid=17131' 2024-08-31T22:19:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=68, tag_tid=8, entry_tid=17220' = 'object_number=71, tag_tid=8, entry_tid=17131' 2024-08-31T22:19:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=68, tag_tid=8, entry_tid=17220' 2024-08-31T22:19:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=68, tag_tid=8, entry_tid=17220' 2024-08-31T22:19:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=68, tag_tid=8, entry_tid=17220' 2024-08-31T22:19:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17220 2024-08-31T22:19:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 17220 2024-08-31T22:19:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=70, tag_tid=8, entry_tid=17314], [object_number=69, tag_tid=8, entry_tid=17313], [object_number=68, tag_tid=8, entry_tid=17312], [object_number=71, tag_tid=8, entry_tid=17311]]], state=connected] 2024-08-31T22:19:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:54.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:54.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:54.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:54.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:54.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=70, tag_tid=8, entry_tid=17314], [object_number=69, tag_tid=8, entry_tid=17313], [object_number=68, tag_tid=8, entry_tid=17312], [object_number=71, tag_tid=8, entry_tid=17311]]], state=connected] 2024-08-31T22:19:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=70, tag_tid=8, entry_tid=17314' 2024-08-31T22:19:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=70, tag_tid=8, entry_tid=17314' 2024-08-31T22:19:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=70, tag_tid=8, entry_tid=17314' '!=' 'object_number=68, tag_tid=8, entry_tid=17220' 2024-08-31T22:19:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=70, tag_tid=8, entry_tid=17314' = 'object_number=68, tag_tid=8, entry_tid=17220' 2024-08-31T22:19:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=70, tag_tid=8, entry_tid=17314' 2024-08-31T22:19:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=70, tag_tid=8, entry_tid=17314' 2024-08-31T22:19:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=70, tag_tid=8, entry_tid=17314' 2024-08-31T22:19:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:55.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17314 2024-08-31T22:19:55.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 17314 2024-08-31T22:19:55.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:55.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:55.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=17419], [object_number=70, tag_tid=8, entry_tid=17418], [object_number=69, tag_tid=8, entry_tid=17417], [object_number=68, tag_tid=8, entry_tid=17416]]], state=connected] 2024-08-31T22:19:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=17419], [object_number=70, tag_tid=8, entry_tid=17418], [object_number=69, tag_tid=8, entry_tid=17417], [object_number=68, tag_tid=8, entry_tid=17416]]], state=connected] 2024-08-31T22:19:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=71, tag_tid=8, entry_tid=17419' 2024-08-31T22:19:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=71, tag_tid=8, entry_tid=17419' 2024-08-31T22:19:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=17419' '!=' 'object_number=70, tag_tid=8, entry_tid=17314' 2024-08-31T22:19:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=17419' = 'object_number=70, tag_tid=8, entry_tid=17314' 2024-08-31T22:19:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=71, tag_tid=8, entry_tid=17419' 2024-08-31T22:19:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=17419' 2024-08-31T22:19:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=17419' 2024-08-31T22:19:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17419 2024-08-31T22:19:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 17419 2024-08-31T22:19:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=69, tag_tid=8, entry_tid=17529], [object_number=68, tag_tid=8, entry_tid=17528], [object_number=71, tag_tid=8, entry_tid=17527], [object_number=70, tag_tid=8, entry_tid=17526]]], state=connected] 2024-08-31T22:19:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=69, tag_tid=8, entry_tid=17529], [object_number=68, tag_tid=8, entry_tid=17528], [object_number=71, tag_tid=8, entry_tid=17527], [object_number=70, tag_tid=8, entry_tid=17526]]], state=connected] 2024-08-31T22:19:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=69, tag_tid=8, entry_tid=17529' 2024-08-31T22:19:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=69, tag_tid=8, entry_tid=17529' 2024-08-31T22:19:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=69, tag_tid=8, entry_tid=17529' '!=' 'object_number=71, tag_tid=8, entry_tid=17419' 2024-08-31T22:19:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=69, tag_tid=8, entry_tid=17529' = 'object_number=71, tag_tid=8, entry_tid=17419' 2024-08-31T22:19:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=69, tag_tid=8, entry_tid=17529' 2024-08-31T22:19:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=69, tag_tid=8, entry_tid=17529' 2024-08-31T22:19:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:55.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=69, tag_tid=8, entry_tid=17529' 2024-08-31T22:19:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17529 2024-08-31T22:19:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 17529 2024-08-31T22:19:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=17623], [object_number=70, tag_tid=8, entry_tid=17622], [object_number=69, tag_tid=8, entry_tid=17621], [object_number=68, tag_tid=8, entry_tid=17620]]], state=connected] 2024-08-31T22:19:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=17623], [object_number=70, tag_tid=8, entry_tid=17622], [object_number=69, tag_tid=8, entry_tid=17621], [object_number=68, tag_tid=8, entry_tid=17620]]], state=connected] 2024-08-31T22:19:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=71, tag_tid=8, entry_tid=17623' 2024-08-31T22:19:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=71, tag_tid=8, entry_tid=17623' 2024-08-31T22:19:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=17623' '!=' 'object_number=69, tag_tid=8, entry_tid=17529' 2024-08-31T22:19:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=17623' = 'object_number=69, tag_tid=8, entry_tid=17529' 2024-08-31T22:19:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=71, tag_tid=8, entry_tid=17623' 2024-08-31T22:19:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=17623' 2024-08-31T22:19:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=17623' 2024-08-31T22:19:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17623 2024-08-31T22:19:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 17623 2024-08-31T22:19:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:56.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:56.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:56.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:56.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:56.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:56.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:56.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=70, tag_tid=8, entry_tid=17714], [object_number=69, tag_tid=8, entry_tid=17713], [object_number=68, tag_tid=8, entry_tid=17712], [object_number=71, tag_tid=8, entry_tid=17711]]], state=connected] 2024-08-31T22:19:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=70, tag_tid=8, entry_tid=17714], [object_number=69, tag_tid=8, entry_tid=17713], [object_number=68, tag_tid=8, entry_tid=17712], [object_number=71, tag_tid=8, entry_tid=17711]]], state=connected] 2024-08-31T22:19:56.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=70, tag_tid=8, entry_tid=17714' 2024-08-31T22:19:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=70, tag_tid=8, entry_tid=17714' 2024-08-31T22:19:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=70, tag_tid=8, entry_tid=17714' '!=' 'object_number=71, tag_tid=8, entry_tid=17623' 2024-08-31T22:19:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=70, tag_tid=8, entry_tid=17714' = 'object_number=71, tag_tid=8, entry_tid=17623' 2024-08-31T22:19:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=70, tag_tid=8, entry_tid=17714' 2024-08-31T22:19:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=70, tag_tid=8, entry_tid=17714' 2024-08-31T22:19:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=70, tag_tid=8, entry_tid=17714' 2024-08-31T22:19:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17714 2024-08-31T22:19:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 17714 2024-08-31T22:19:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=70, tag_tid=8, entry_tid=17810], [object_number=69, tag_tid=8, entry_tid=17809], [object_number=68, tag_tid=8, entry_tid=17808], [object_number=71, tag_tid=8, entry_tid=17807]]], state=connected] 2024-08-31T22:19:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=70, tag_tid=8, entry_tid=17810], [object_number=69, tag_tid=8, entry_tid=17809], [object_number=68, tag_tid=8, entry_tid=17808], [object_number=71, tag_tid=8, entry_tid=17807]]], state=connected] 2024-08-31T22:19:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=70, tag_tid=8, entry_tid=17810' 2024-08-31T22:19:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=70, tag_tid=8, entry_tid=17810' 2024-08-31T22:19:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=70, tag_tid=8, entry_tid=17810' '!=' 'object_number=70, tag_tid=8, entry_tid=17714' 2024-08-31T22:19:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=70, tag_tid=8, entry_tid=17810' = 'object_number=70, tag_tid=8, entry_tid=17714' 2024-08-31T22:19:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=70, tag_tid=8, entry_tid=17810' 2024-08-31T22:19:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=70, tag_tid=8, entry_tid=17810' 2024-08-31T22:19:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=70, tag_tid=8, entry_tid=17810' 2024-08-31T22:19:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17810 2024-08-31T22:19:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 17810 2024-08-31T22:19:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=70, tag_tid=8, entry_tid=17910], [object_number=69, tag_tid=8, entry_tid=17909], [object_number=68, tag_tid=8, entry_tid=17908], [object_number=71, tag_tid=8, entry_tid=17907]]], state=connected] 2024-08-31T22:19:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:57.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=70, tag_tid=8, entry_tid=17910], [object_number=69, tag_tid=8, entry_tid=17909], [object_number=68, tag_tid=8, entry_tid=17908], [object_number=71, tag_tid=8, entry_tid=17907]]], state=connected] 2024-08-31T22:19:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=70, tag_tid=8, entry_tid=17910' 2024-08-31T22:19:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=70, tag_tid=8, entry_tid=17910' 2024-08-31T22:19:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=70, tag_tid=8, entry_tid=17910' '!=' 'object_number=70, tag_tid=8, entry_tid=17810' 2024-08-31T22:19:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=70, tag_tid=8, entry_tid=17910' = 'object_number=70, tag_tid=8, entry_tid=17810' 2024-08-31T22:19:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=70, tag_tid=8, entry_tid=17910' 2024-08-31T22:19:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=70, tag_tid=8, entry_tid=17910' 2024-08-31T22:19:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=70, tag_tid=8, entry_tid=17910' 2024-08-31T22:19:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17910 2024-08-31T22:19:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 17910 2024-08-31T22:19:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:58.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:58.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:58.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:58.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:58.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:58.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=18011], [object_number=70, tag_tid=8, entry_tid=18010], [object_number=69, tag_tid=8, entry_tid=18009], [object_number=68, tag_tid=8, entry_tid=18008]]], state=connected] 2024-08-31T22:19:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=18011], [object_number=70, tag_tid=8, entry_tid=18010], [object_number=69, tag_tid=8, entry_tid=18009], [object_number=68, tag_tid=8, entry_tid=18008]]], state=connected] 2024-08-31T22:19:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=71, tag_tid=8, entry_tid=18011' 2024-08-31T22:19:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=71, tag_tid=8, entry_tid=18011' 2024-08-31T22:19:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=18011' '!=' 'object_number=70, tag_tid=8, entry_tid=17910' 2024-08-31T22:19:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=18011' = 'object_number=70, tag_tid=8, entry_tid=17910' 2024-08-31T22:19:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=71, tag_tid=8, entry_tid=18011' 2024-08-31T22:19:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=18011' 2024-08-31T22:19:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=18011' 2024-08-31T22:19:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18011 2024-08-31T22:19:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 18011 2024-08-31T22:19:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=70, tag_tid=8, entry_tid=18102], [object_number=69, tag_tid=8, entry_tid=18101], [object_number=68, tag_tid=8, entry_tid=18100], [object_number=71, tag_tid=8, entry_tid=18099]]], state=connected] 2024-08-31T22:19:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=70, tag_tid=8, entry_tid=18102], [object_number=69, tag_tid=8, entry_tid=18101], [object_number=68, tag_tid=8, entry_tid=18100], [object_number=71, tag_tid=8, entry_tid=18099]]], state=connected] 2024-08-31T22:19:59.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=70, tag_tid=8, entry_tid=18102' 2024-08-31T22:19:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=70, tag_tid=8, entry_tid=18102' 2024-08-31T22:19:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=70, tag_tid=8, entry_tid=18102' '!=' 'object_number=71, tag_tid=8, entry_tid=18011' 2024-08-31T22:19:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=70, tag_tid=8, entry_tid=18102' = 'object_number=71, tag_tid=8, entry_tid=18011' 2024-08-31T22:19:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=70, tag_tid=8, entry_tid=18102' 2024-08-31T22:19:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=70, tag_tid=8, entry_tid=18102' 2024-08-31T22:19:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=70, tag_tid=8, entry_tid=18102' 2024-08-31T22:19:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:59.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18102 2024-08-31T22:19:59.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 18102 2024-08-31T22:19:59.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:59.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:59.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:19:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:19:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:19:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:19:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:19:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=68, tag_tid=8, entry_tid=18244], [object_number=71, tag_tid=8, entry_tid=18243], [object_number=70, tag_tid=8, entry_tid=18242], [object_number=69, tag_tid=8, entry_tid=18241]]], state=connected] 2024-08-31T22:19:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:19:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:19:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:19:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:19:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:19:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:19:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:19:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:19:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:19:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:19:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:19:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:19:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:19:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:19:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=68, tag_tid=8, entry_tid=18244], [object_number=71, tag_tid=8, entry_tid=18243], [object_number=70, tag_tid=8, entry_tid=18242], [object_number=69, tag_tid=8, entry_tid=18241]]], state=connected] 2024-08-31T22:19:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:19:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=68, tag_tid=8, entry_tid=18244' 2024-08-31T22:19:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=68, tag_tid=8, entry_tid=18244' 2024-08-31T22:19:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=68, tag_tid=8, entry_tid=18244' '!=' 'object_number=70, tag_tid=8, entry_tid=18102' 2024-08-31T22:19:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:19:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=68, tag_tid=8, entry_tid=18244' = 'object_number=70, tag_tid=8, entry_tid=18102' 2024-08-31T22:19:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=68, tag_tid=8, entry_tid=18244' 2024-08-31T22:19:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:19:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=68, tag_tid=8, entry_tid=18244' 2024-08-31T22:19:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:19:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:19:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:19:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:19:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=68, tag_tid=8, entry_tid=18244' 2024-08-31T22:19:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:19:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:19:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18244 2024-08-31T22:19:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 18244 2024-08-31T22:19:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:19:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:19:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:19:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:19:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:19:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:19:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:19:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:19:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:19:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:19:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:19:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:19:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:19:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:19:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:19:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:19:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:19:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:19:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:19:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:19:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:19:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:19:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:19:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:19:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:19:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:19:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:19:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:19:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:19:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:19:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:19:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:19:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:19:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:19:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:19:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:19:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:19:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:19:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:19:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:19:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:19:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:19:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=68, tag_tid=8, entry_tid=18344], [object_number=71, tag_tid=8, entry_tid=18343], [object_number=70, tag_tid=8, entry_tid=18342], [object_number=69, tag_tid=8, entry_tid=18341]]], state=connected] 2024-08-31T22:20:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:00.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:00.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:00.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:00.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:00.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=68, tag_tid=8, entry_tid=18344], [object_number=71, tag_tid=8, entry_tid=18343], [object_number=70, tag_tid=8, entry_tid=18342], [object_number=69, tag_tid=8, entry_tid=18341]]], state=connected] 2024-08-31T22:20:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=68, tag_tid=8, entry_tid=18344' 2024-08-31T22:20:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=68, tag_tid=8, entry_tid=18344' 2024-08-31T22:20:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=68, tag_tid=8, entry_tid=18344' '!=' 'object_number=68, tag_tid=8, entry_tid=18244' 2024-08-31T22:20:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=68, tag_tid=8, entry_tid=18344' = 'object_number=68, tag_tid=8, entry_tid=18244' 2024-08-31T22:20:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=68, tag_tid=8, entry_tid=18344' 2024-08-31T22:20:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=68, tag_tid=8, entry_tid=18344' 2024-08-31T22:20:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=68, tag_tid=8, entry_tid=18344' 2024-08-31T22:20:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18344 2024-08-31T22:20:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 18344 2024-08-31T22:20:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=68, tag_tid=8, entry_tid=18448], [object_number=71, tag_tid=8, entry_tid=18447], [object_number=70, tag_tid=8, entry_tid=18446], [object_number=69, tag_tid=8, entry_tid=18445]]], state=connected] 2024-08-31T22:20:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=68, tag_tid=8, entry_tid=18448], [object_number=71, tag_tid=8, entry_tid=18447], [object_number=70, tag_tid=8, entry_tid=18446], [object_number=69, tag_tid=8, entry_tid=18445]]], state=connected] 2024-08-31T22:20:00.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=68, tag_tid=8, entry_tid=18448' 2024-08-31T22:20:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=68, tag_tid=8, entry_tid=18448' 2024-08-31T22:20:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=68, tag_tid=8, entry_tid=18448' '!=' 'object_number=68, tag_tid=8, entry_tid=18344' 2024-08-31T22:20:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=68, tag_tid=8, entry_tid=18448' = 'object_number=68, tag_tid=8, entry_tid=18344' 2024-08-31T22:20:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=68, tag_tid=8, entry_tid=18448' 2024-08-31T22:20:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:00.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=68, tag_tid=8, entry_tid=18448' 2024-08-31T22:20:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=68, tag_tid=8, entry_tid=18448' 2024-08-31T22:20:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18448 2024-08-31T22:20:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 18448 2024-08-31T22:20:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:01.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:01.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:01.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:01.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:01.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:01.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:01.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:01.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:01.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:01.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=70, tag_tid=8, entry_tid=18582], [object_number=69, tag_tid=8, entry_tid=18581], [object_number=68, tag_tid=8, entry_tid=18580], [object_number=71, tag_tid=8, entry_tid=18579]]], state=connected] 2024-08-31T22:20:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=70, tag_tid=8, entry_tid=18582], [object_number=69, tag_tid=8, entry_tid=18581], [object_number=68, tag_tid=8, entry_tid=18580], [object_number=71, tag_tid=8, entry_tid=18579]]], state=connected] 2024-08-31T22:20:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=70, tag_tid=8, entry_tid=18582' 2024-08-31T22:20:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=70, tag_tid=8, entry_tid=18582' 2024-08-31T22:20:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=70, tag_tid=8, entry_tid=18582' '!=' 'object_number=68, tag_tid=8, entry_tid=18448' 2024-08-31T22:20:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=70, tag_tid=8, entry_tid=18582' = 'object_number=68, tag_tid=8, entry_tid=18448' 2024-08-31T22:20:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=70, tag_tid=8, entry_tid=18582' 2024-08-31T22:20:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=70, tag_tid=8, entry_tid=18582' 2024-08-31T22:20:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=70, tag_tid=8, entry_tid=18582' 2024-08-31T22:20:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18582 2024-08-31T22:20:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 18582 2024-08-31T22:20:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=70, tag_tid=8, entry_tid=18710], [object_number=69, tag_tid=8, entry_tid=18709], [object_number=68, tag_tid=8, entry_tid=18708], [object_number=71, tag_tid=8, entry_tid=18707]]], state=connected] 2024-08-31T22:20:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=70, tag_tid=8, entry_tid=18710], [object_number=69, tag_tid=8, entry_tid=18709], [object_number=68, tag_tid=8, entry_tid=18708], [object_number=71, tag_tid=8, entry_tid=18707]]], state=connected] 2024-08-31T22:20:02.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=70, tag_tid=8, entry_tid=18710' 2024-08-31T22:20:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=70, tag_tid=8, entry_tid=18710' 2024-08-31T22:20:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=70, tag_tid=8, entry_tid=18710' '!=' 'object_number=70, tag_tid=8, entry_tid=18582' 2024-08-31T22:20:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=70, tag_tid=8, entry_tid=18710' = 'object_number=70, tag_tid=8, entry_tid=18582' 2024-08-31T22:20:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=70, tag_tid=8, entry_tid=18710' 2024-08-31T22:20:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=70, tag_tid=8, entry_tid=18710' 2024-08-31T22:20:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=70, tag_tid=8, entry_tid=18710' 2024-08-31T22:20:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18710 2024-08-31T22:20:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 18710 2024-08-31T22:20:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:02.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:02.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:02.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:02.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:02.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=18799], [object_number=70, tag_tid=8, entry_tid=18798], [object_number=69, tag_tid=8, entry_tid=18797], [object_number=68, tag_tid=8, entry_tid=18796]]], state=connected] 2024-08-31T22:20:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:02.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:02.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:02.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:02.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=18799], [object_number=70, tag_tid=8, entry_tid=18798], [object_number=69, tag_tid=8, entry_tid=18797], [object_number=68, tag_tid=8, entry_tid=18796]]], state=connected] 2024-08-31T22:20:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=71, tag_tid=8, entry_tid=18799' 2024-08-31T22:20:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=71, tag_tid=8, entry_tid=18799' 2024-08-31T22:20:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=18799' '!=' 'object_number=70, tag_tid=8, entry_tid=18710' 2024-08-31T22:20:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=18799' = 'object_number=70, tag_tid=8, entry_tid=18710' 2024-08-31T22:20:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=71, tag_tid=8, entry_tid=18799' 2024-08-31T22:20:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:02.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=18799' 2024-08-31T22:20:02.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:02.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=18799' 2024-08-31T22:20:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18799 2024-08-31T22:20:02.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 18799 2024-08-31T22:20:02.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:02.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:02.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=18883], [object_number=70, tag_tid=8, entry_tid=18882], [object_number=69, tag_tid=8, entry_tid=18881], [object_number=68, tag_tid=8, entry_tid=18880]]], state=connected] 2024-08-31T22:20:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:02.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:02.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:02.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:02.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:02.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=18883], [object_number=70, tag_tid=8, entry_tid=18882], [object_number=69, tag_tid=8, entry_tid=18881], [object_number=68, tag_tid=8, entry_tid=18880]]], state=connected] 2024-08-31T22:20:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=71, tag_tid=8, entry_tid=18883' 2024-08-31T22:20:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=71, tag_tid=8, entry_tid=18883' 2024-08-31T22:20:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=18883' '!=' 'object_number=71, tag_tid=8, entry_tid=18799' 2024-08-31T22:20:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=18883' = 'object_number=71, tag_tid=8, entry_tid=18799' 2024-08-31T22:20:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=71, tag_tid=8, entry_tid=18883' 2024-08-31T22:20:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=18883' 2024-08-31T22:20:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=18883' 2024-08-31T22:20:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:03.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18883 2024-08-31T22:20:03.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 18883 2024-08-31T22:20:03.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:03.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:03.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=68, tag_tid=8, entry_tid=19012], [object_number=71, tag_tid=8, entry_tid=19011], [object_number=70, tag_tid=8, entry_tid=19010], [object_number=69, tag_tid=8, entry_tid=19009]]], state=connected] 2024-08-31T22:20:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=68, tag_tid=8, entry_tid=19012], [object_number=71, tag_tid=8, entry_tid=19011], [object_number=70, tag_tid=8, entry_tid=19010], [object_number=69, tag_tid=8, entry_tid=19009]]], state=connected] 2024-08-31T22:20:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=68, tag_tid=8, entry_tid=19012' 2024-08-31T22:20:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=68, tag_tid=8, entry_tid=19012' 2024-08-31T22:20:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=68, tag_tid=8, entry_tid=19012' '!=' 'object_number=71, tag_tid=8, entry_tid=18883' 2024-08-31T22:20:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=68, tag_tid=8, entry_tid=19012' = 'object_number=71, tag_tid=8, entry_tid=18883' 2024-08-31T22:20:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=68, tag_tid=8, entry_tid=19012' 2024-08-31T22:20:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=68, tag_tid=8, entry_tid=19012' 2024-08-31T22:20:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=68, tag_tid=8, entry_tid=19012' 2024-08-31T22:20:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19012 2024-08-31T22:20:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 19012 2024-08-31T22:20:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:04.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:04.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=19107], [object_number=70, tag_tid=8, entry_tid=19106], [object_number=69, tag_tid=8, entry_tid=19105], [object_number=68, tag_tid=8, entry_tid=19104]]], state=connected] 2024-08-31T22:20:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:04.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:04.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:04.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:04.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:04.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=19107], [object_number=70, tag_tid=8, entry_tid=19106], [object_number=69, tag_tid=8, entry_tid=19105], [object_number=68, tag_tid=8, entry_tid=19104]]], state=connected] 2024-08-31T22:20:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=71, tag_tid=8, entry_tid=19107' 2024-08-31T22:20:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=71, tag_tid=8, entry_tid=19107' 2024-08-31T22:20:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=19107' '!=' 'object_number=68, tag_tid=8, entry_tid=19012' 2024-08-31T22:20:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=19107' = 'object_number=68, tag_tid=8, entry_tid=19012' 2024-08-31T22:20:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=71, tag_tid=8, entry_tid=19107' 2024-08-31T22:20:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=19107' 2024-08-31T22:20:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=19107' 2024-08-31T22:20:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19107 2024-08-31T22:20:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 19107 2024-08-31T22:20:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:04.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:04.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:04.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:04.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:04.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:04.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=69, tag_tid=8, entry_tid=19201], [object_number=68, tag_tid=8, entry_tid=19200], [object_number=71, tag_tid=8, entry_tid=19199], [object_number=70, tag_tid=8, entry_tid=19198]]], state=connected] 2024-08-31T22:20:04.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=69, tag_tid=8, entry_tid=19201], [object_number=68, tag_tid=8, entry_tid=19200], [object_number=71, tag_tid=8, entry_tid=19199], [object_number=70, tag_tid=8, entry_tid=19198]]], state=connected] 2024-08-31T22:20:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=69, tag_tid=8, entry_tid=19201' 2024-08-31T22:20:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=69, tag_tid=8, entry_tid=19201' 2024-08-31T22:20:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=69, tag_tid=8, entry_tid=19201' '!=' 'object_number=71, tag_tid=8, entry_tid=19107' 2024-08-31T22:20:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=69, tag_tid=8, entry_tid=19201' = 'object_number=71, tag_tid=8, entry_tid=19107' 2024-08-31T22:20:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=69, tag_tid=8, entry_tid=19201' 2024-08-31T22:20:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=69, tag_tid=8, entry_tid=19201' 2024-08-31T22:20:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=69, tag_tid=8, entry_tid=19201' 2024-08-31T22:20:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19201 2024-08-31T22:20:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 19201 2024-08-31T22:20:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:04.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=70, tag_tid=8, entry_tid=19306], [object_number=69, tag_tid=8, entry_tid=19305], [object_number=68, tag_tid=8, entry_tid=19304], [object_number=71, tag_tid=8, entry_tid=19303]]], state=connected] 2024-08-31T22:20:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=70, tag_tid=8, entry_tid=19306], [object_number=69, tag_tid=8, entry_tid=19305], [object_number=68, tag_tid=8, entry_tid=19304], [object_number=71, tag_tid=8, entry_tid=19303]]], state=connected] 2024-08-31T22:20:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=70, tag_tid=8, entry_tid=19306' 2024-08-31T22:20:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=70, tag_tid=8, entry_tid=19306' 2024-08-31T22:20:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=70, tag_tid=8, entry_tid=19306' '!=' 'object_number=69, tag_tid=8, entry_tid=19201' 2024-08-31T22:20:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=70, tag_tid=8, entry_tid=19306' = 'object_number=69, tag_tid=8, entry_tid=19201' 2024-08-31T22:20:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=70, tag_tid=8, entry_tid=19306' 2024-08-31T22:20:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=70, tag_tid=8, entry_tid=19306' 2024-08-31T22:20:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=70, tag_tid=8, entry_tid=19306' 2024-08-31T22:20:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19306 2024-08-31T22:20:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 19306 2024-08-31T22:20:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:05.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:05.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:05.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=68, tag_tid=8, entry_tid=19408], [object_number=71, tag_tid=8, entry_tid=19407], [object_number=70, tag_tid=8, entry_tid=19406], [object_number=69, tag_tid=8, entry_tid=19405]]], state=connected] 2024-08-31T22:20:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=68, tag_tid=8, entry_tid=19408], [object_number=71, tag_tid=8, entry_tid=19407], [object_number=70, tag_tid=8, entry_tid=19406], [object_number=69, tag_tid=8, entry_tid=19405]]], state=connected] 2024-08-31T22:20:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=68, tag_tid=8, entry_tid=19408' 2024-08-31T22:20:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=68, tag_tid=8, entry_tid=19408' 2024-08-31T22:20:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=68, tag_tid=8, entry_tid=19408' '!=' 'object_number=70, tag_tid=8, entry_tid=19306' 2024-08-31T22:20:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=68, tag_tid=8, entry_tid=19408' = 'object_number=70, tag_tid=8, entry_tid=19306' 2024-08-31T22:20:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=68, tag_tid=8, entry_tid=19408' 2024-08-31T22:20:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=68, tag_tid=8, entry_tid=19408' 2024-08-31T22:20:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=68, tag_tid=8, entry_tid=19408' 2024-08-31T22:20:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19408 2024-08-31T22:20:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 19408 2024-08-31T22:20:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=69, tag_tid=8, entry_tid=19517], [object_number=68, tag_tid=8, entry_tid=19516], [object_number=71, tag_tid=8, entry_tid=19515], [object_number=70, tag_tid=8, entry_tid=19514]]], state=connected] 2024-08-31T22:20:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=69, tag_tid=8, entry_tid=19517], [object_number=68, tag_tid=8, entry_tid=19516], [object_number=71, tag_tid=8, entry_tid=19515], [object_number=70, tag_tid=8, entry_tid=19514]]], state=connected] 2024-08-31T22:20:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=69, tag_tid=8, entry_tid=19517' 2024-08-31T22:20:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=69, tag_tid=8, entry_tid=19517' 2024-08-31T22:20:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=69, tag_tid=8, entry_tid=19517' '!=' 'object_number=68, tag_tid=8, entry_tid=19408' 2024-08-31T22:20:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=69, tag_tid=8, entry_tid=19517' = 'object_number=68, tag_tid=8, entry_tid=19408' 2024-08-31T22:20:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=69, tag_tid=8, entry_tid=19517' 2024-08-31T22:20:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=69, tag_tid=8, entry_tid=19517' 2024-08-31T22:20:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=69, tag_tid=8, entry_tid=19517' 2024-08-31T22:20:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19517 2024-08-31T22:20:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 19517 2024-08-31T22:20:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:06.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=19623], [object_number=70, tag_tid=8, entry_tid=19622], [object_number=69, tag_tid=8, entry_tid=19621], [object_number=68, tag_tid=8, entry_tid=19620]]], state=connected] 2024-08-31T22:20:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:06.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=19623], [object_number=70, tag_tid=8, entry_tid=19622], [object_number=69, tag_tid=8, entry_tid=19621], [object_number=68, tag_tid=8, entry_tid=19620]]], state=connected] 2024-08-31T22:20:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=71, tag_tid=8, entry_tid=19623' 2024-08-31T22:20:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=71, tag_tid=8, entry_tid=19623' 2024-08-31T22:20:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=19623' '!=' 'object_number=69, tag_tid=8, entry_tid=19517' 2024-08-31T22:20:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=19623' = 'object_number=69, tag_tid=8, entry_tid=19517' 2024-08-31T22:20:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=71, tag_tid=8, entry_tid=19623' 2024-08-31T22:20:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=19623' 2024-08-31T22:20:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=19623' 2024-08-31T22:20:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19623 2024-08-31T22:20:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 19623 2024-08-31T22:20:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:07.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=68, tag_tid=8, entry_tid=19720], [object_number=71, tag_tid=8, entry_tid=19719], [object_number=70, tag_tid=8, entry_tid=19718], [object_number=69, tag_tid=8, entry_tid=19717]]], state=connected] 2024-08-31T22:20:07.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:07.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:07.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:07.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:07.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:07.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:07.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:07.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:07.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:07.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:07.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:07.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:07.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:07.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=68, tag_tid=8, entry_tid=19720], [object_number=71, tag_tid=8, entry_tid=19719], [object_number=70, tag_tid=8, entry_tid=19718], [object_number=69, tag_tid=8, entry_tid=19717]]], state=connected] 2024-08-31T22:20:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=68, tag_tid=8, entry_tid=19720' 2024-08-31T22:20:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=68, tag_tid=8, entry_tid=19720' 2024-08-31T22:20:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=68, tag_tid=8, entry_tid=19720' '!=' 'object_number=71, tag_tid=8, entry_tid=19623' 2024-08-31T22:20:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=68, tag_tid=8, entry_tid=19720' = 'object_number=71, tag_tid=8, entry_tid=19623' 2024-08-31T22:20:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=68, tag_tid=8, entry_tid=19720' 2024-08-31T22:20:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=68, tag_tid=8, entry_tid=19720' 2024-08-31T22:20:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=68, tag_tid=8, entry_tid=19720' 2024-08-31T22:20:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19720 2024-08-31T22:20:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 19720 2024-08-31T22:20:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:07.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:07.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=19859], [object_number=70, tag_tid=8, entry_tid=19858], [object_number=69, tag_tid=8, entry_tid=19857], [object_number=68, tag_tid=8, entry_tid=19856]]], state=connected] 2024-08-31T22:20:07.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=71, tag_tid=8, entry_tid=19859], [object_number=70, tag_tid=8, entry_tid=19858], [object_number=69, tag_tid=8, entry_tid=19857], [object_number=68, tag_tid=8, entry_tid=19856]]], state=connected] 2024-08-31T22:20:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=71, tag_tid=8, entry_tid=19859' 2024-08-31T22:20:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=71, tag_tid=8, entry_tid=19859' 2024-08-31T22:20:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=19859' '!=' 'object_number=68, tag_tid=8, entry_tid=19720' 2024-08-31T22:20:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=71, tag_tid=8, entry_tid=19859' = 'object_number=68, tag_tid=8, entry_tid=19720' 2024-08-31T22:20:07.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=71, tag_tid=8, entry_tid=19859' 2024-08-31T22:20:07.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:07.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:07.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=19859' 2024-08-31T22:20:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:07.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=71, tag_tid=8, entry_tid=19859' 2024-08-31T22:20:07.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:07.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19859 2024-08-31T22:20:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 19859 2024-08-31T22:20:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=69, tag_tid=8, entry_tid=19957], [object_number=68, tag_tid=8, entry_tid=19956], [object_number=71, tag_tid=8, entry_tid=19955], [object_number=70, tag_tid=8, entry_tid=19954]]], state=connected] 2024-08-31T22:20:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=69, tag_tid=8, entry_tid=19957], [object_number=68, tag_tid=8, entry_tid=19956], [object_number=71, tag_tid=8, entry_tid=19955], [object_number=70, tag_tid=8, entry_tid=19954]]], state=connected] 2024-08-31T22:20:08.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=69, tag_tid=8, entry_tid=19957' 2024-08-31T22:20:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=69, tag_tid=8, entry_tid=19957' 2024-08-31T22:20:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=69, tag_tid=8, entry_tid=19957' '!=' 'object_number=71, tag_tid=8, entry_tid=19859' 2024-08-31T22:20:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=69, tag_tid=8, entry_tid=19957' = 'object_number=71, tag_tid=8, entry_tid=19859' 2024-08-31T22:20:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=69, tag_tid=8, entry_tid=19957' 2024-08-31T22:20:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=69, tag_tid=8, entry_tid=19957' 2024-08-31T22:20:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=69, tag_tid=8, entry_tid=19957' 2024-08-31T22:20:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19957 2024-08-31T22:20:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 19957 2024-08-31T22:20:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:08.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:08.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:08.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:08.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=70, tag_tid=8, entry_tid=20062], [object_number=69, tag_tid=8, entry_tid=20061], [object_number=68, tag_tid=8, entry_tid=20060], [object_number=71, tag_tid=8, entry_tid=20059]]], state=connected] 2024-08-31T22:20:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:08.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:08.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:08.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:08.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:08.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=70, tag_tid=8, entry_tid=20062], [object_number=69, tag_tid=8, entry_tid=20061], [object_number=68, tag_tid=8, entry_tid=20060], [object_number=71, tag_tid=8, entry_tid=20059]]], state=connected] 2024-08-31T22:20:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=70, tag_tid=8, entry_tid=20062' 2024-08-31T22:20:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=70, tag_tid=8, entry_tid=20062' 2024-08-31T22:20:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=70, tag_tid=8, entry_tid=20062' '!=' 'object_number=69, tag_tid=8, entry_tid=19957' 2024-08-31T22:20:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=70, tag_tid=8, entry_tid=20062' = 'object_number=69, tag_tid=8, entry_tid=19957' 2024-08-31T22:20:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=70, tag_tid=8, entry_tid=20062' 2024-08-31T22:20:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=70, tag_tid=8, entry_tid=20062' 2024-08-31T22:20:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=70, tag_tid=8, entry_tid=20062' 2024-08-31T22:20:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20062 2024-08-31T22:20:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 20062 2024-08-31T22:20:08.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:08.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:08.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:09.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:09.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=69, tag_tid=8, entry_tid=20157], [object_number=68, tag_tid=8, entry_tid=20156], [object_number=71, tag_tid=8, entry_tid=20155], [object_number=70, tag_tid=8, entry_tid=20154]]], state=connected] 2024-08-31T22:20:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=69, tag_tid=8, entry_tid=20157], [object_number=68, tag_tid=8, entry_tid=20156], [object_number=71, tag_tid=8, entry_tid=20155], [object_number=70, tag_tid=8, entry_tid=20154]]], state=connected] 2024-08-31T22:20:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=69, tag_tid=8, entry_tid=20157' 2024-08-31T22:20:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=69, tag_tid=8, entry_tid=20157' 2024-08-31T22:20:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=69, tag_tid=8, entry_tid=20157' '!=' 'object_number=70, tag_tid=8, entry_tid=20062' 2024-08-31T22:20:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=69, tag_tid=8, entry_tid=20157' = 'object_number=70, tag_tid=8, entry_tid=20062' 2024-08-31T22:20:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=69, tag_tid=8, entry_tid=20157' 2024-08-31T22:20:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=69, tag_tid=8, entry_tid=20157' 2024-08-31T22:20:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=69, tag_tid=8, entry_tid=20157' 2024-08-31T22:20:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20157 2024-08-31T22:20:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 20157 2024-08-31T22:20:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:09.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:09.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:09.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:09.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:09.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=69, tag_tid=8, entry_tid=20257], [object_number=68, tag_tid=8, entry_tid=20256], [object_number=71, tag_tid=8, entry_tid=20255], [object_number=70, tag_tid=8, entry_tid=20254]]], state=connected] 2024-08-31T22:20:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:09.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 17 2024-08-31T22:20:09.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:09.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:09.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:09.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=69, tag_tid=8, entry_tid=20257], [object_number=68, tag_tid=8, entry_tid=20256], [object_number=71, tag_tid=8, entry_tid=20255], [object_number=70, tag_tid=8, entry_tid=20254]]], state=connected] 2024-08-31T22:20:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=69, tag_tid=8, entry_tid=20257' 2024-08-31T22:20:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=69, tag_tid=8, entry_tid=20257' 2024-08-31T22:20:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=69, tag_tid=8, entry_tid=20257' '!=' 'object_number=69, tag_tid=8, entry_tid=20157' 2024-08-31T22:20:09.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:09.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=69, tag_tid=8, entry_tid=20257' = 'object_number=69, tag_tid=8, entry_tid=20157' 2024-08-31T22:20:09.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=69, tag_tid=8, entry_tid=20257' 2024-08-31T22:20:09.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:09.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:09.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:09.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=69, tag_tid=8, entry_tid=20257' 2024-08-31T22:20:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=69, tag_tid=8, entry_tid=20257' 2024-08-31T22:20:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:09.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20257 2024-08-31T22:20:09.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 20257 2024-08-31T22:20:09.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:09.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:09.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:10.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:10.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=74, tag_tid=8, entry_tid=20346], [object_number=73, tag_tid=8, entry_tid=20345], [object_number=72, tag_tid=8, entry_tid=20344], [object_number=75, tag_tid=8, entry_tid=20343]]], state=connected] 2024-08-31T22:20:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=74, tag_tid=8, entry_tid=20346], [object_number=73, tag_tid=8, entry_tid=20345], [object_number=72, tag_tid=8, entry_tid=20344], [object_number=75, tag_tid=8, entry_tid=20343]]], state=connected] 2024-08-31T22:20:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=74, tag_tid=8, entry_tid=20346' 2024-08-31T22:20:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=74, tag_tid=8, entry_tid=20346' 2024-08-31T22:20:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=74, tag_tid=8, entry_tid=20346' '!=' 'object_number=69, tag_tid=8, entry_tid=20257' 2024-08-31T22:20:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=74, tag_tid=8, entry_tid=20346' = 'object_number=69, tag_tid=8, entry_tid=20257' 2024-08-31T22:20:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=74, tag_tid=8, entry_tid=20346' 2024-08-31T22:20:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=74, tag_tid=8, entry_tid=20346' 2024-08-31T22:20:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=74, tag_tid=8, entry_tid=20346' 2024-08-31T22:20:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20346 2024-08-31T22:20:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 20346 2024-08-31T22:20:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=75, tag_tid=8, entry_tid=20519], [object_number=74, tag_tid=8, entry_tid=20518], [object_number=73, tag_tid=8, entry_tid=20517], [object_number=72, tag_tid=8, entry_tid=20516]]], state=connected] 2024-08-31T22:20:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=75, tag_tid=8, entry_tid=20519], [object_number=74, tag_tid=8, entry_tid=20518], [object_number=73, tag_tid=8, entry_tid=20517], [object_number=72, tag_tid=8, entry_tid=20516]]], state=connected] 2024-08-31T22:20:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=75, tag_tid=8, entry_tid=20519' 2024-08-31T22:20:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=75, tag_tid=8, entry_tid=20519' 2024-08-31T22:20:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=75, tag_tid=8, entry_tid=20519' '!=' 'object_number=74, tag_tid=8, entry_tid=20346' 2024-08-31T22:20:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=75, tag_tid=8, entry_tid=20519' = 'object_number=74, tag_tid=8, entry_tid=20346' 2024-08-31T22:20:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=75, tag_tid=8, entry_tid=20519' 2024-08-31T22:20:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=75, tag_tid=8, entry_tid=20519' 2024-08-31T22:20:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=75, tag_tid=8, entry_tid=20519' 2024-08-31T22:20:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20519 2024-08-31T22:20:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 20519 2024-08-31T22:20:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:11.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:11.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:11.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:11.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:11.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:11.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:11.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:11.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=72, tag_tid=8, entry_tid=20632], [object_number=75, tag_tid=8, entry_tid=20631], [object_number=74, tag_tid=8, entry_tid=20630], [object_number=73, tag_tid=8, entry_tid=20629]]], state=connected] 2024-08-31T22:20:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=72, tag_tid=8, entry_tid=20632], [object_number=75, tag_tid=8, entry_tid=20631], [object_number=74, tag_tid=8, entry_tid=20630], [object_number=73, tag_tid=8, entry_tid=20629]]], state=connected] 2024-08-31T22:20:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=72, tag_tid=8, entry_tid=20632' 2024-08-31T22:20:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=72, tag_tid=8, entry_tid=20632' 2024-08-31T22:20:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=72, tag_tid=8, entry_tid=20632' '!=' 'object_number=75, tag_tid=8, entry_tid=20519' 2024-08-31T22:20:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=72, tag_tid=8, entry_tid=20632' = 'object_number=75, tag_tid=8, entry_tid=20519' 2024-08-31T22:20:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=72, tag_tid=8, entry_tid=20632' 2024-08-31T22:20:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=72, tag_tid=8, entry_tid=20632' 2024-08-31T22:20:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=72, tag_tid=8, entry_tid=20632' 2024-08-31T22:20:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20632 2024-08-31T22:20:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 20632 2024-08-31T22:20:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=75, tag_tid=8, entry_tid=20739], [object_number=74, tag_tid=8, entry_tid=20738], [object_number=73, tag_tid=8, entry_tid=20737], [object_number=72, tag_tid=8, entry_tid=20736]]], state=connected] 2024-08-31T22:20:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:12.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=75, tag_tid=8, entry_tid=20739], [object_number=74, tag_tid=8, entry_tid=20738], [object_number=73, tag_tid=8, entry_tid=20737], [object_number=72, tag_tid=8, entry_tid=20736]]], state=connected] 2024-08-31T22:20:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=75, tag_tid=8, entry_tid=20739' 2024-08-31T22:20:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=75, tag_tid=8, entry_tid=20739' 2024-08-31T22:20:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=75, tag_tid=8, entry_tid=20739' '!=' 'object_number=72, tag_tid=8, entry_tid=20632' 2024-08-31T22:20:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=75, tag_tid=8, entry_tid=20739' = 'object_number=72, tag_tid=8, entry_tid=20632' 2024-08-31T22:20:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=75, tag_tid=8, entry_tid=20739' 2024-08-31T22:20:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=75, tag_tid=8, entry_tid=20739' 2024-08-31T22:20:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=75, tag_tid=8, entry_tid=20739' 2024-08-31T22:20:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20739 2024-08-31T22:20:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 20739 2024-08-31T22:20:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=73, tag_tid=8, entry_tid=20849], [object_number=72, tag_tid=8, entry_tid=20848], [object_number=75, tag_tid=8, entry_tid=20847], [object_number=74, tag_tid=8, entry_tid=20846]]], state=connected] 2024-08-31T22:20:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=73, tag_tid=8, entry_tid=20849], [object_number=72, tag_tid=8, entry_tid=20848], [object_number=75, tag_tid=8, entry_tid=20847], [object_number=74, tag_tid=8, entry_tid=20846]]], state=connected] 2024-08-31T22:20:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=73, tag_tid=8, entry_tid=20849' 2024-08-31T22:20:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=73, tag_tid=8, entry_tid=20849' 2024-08-31T22:20:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=73, tag_tid=8, entry_tid=20849' '!=' 'object_number=75, tag_tid=8, entry_tid=20739' 2024-08-31T22:20:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=73, tag_tid=8, entry_tid=20849' = 'object_number=75, tag_tid=8, entry_tid=20739' 2024-08-31T22:20:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=73, tag_tid=8, entry_tid=20849' 2024-08-31T22:20:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=8, entry_tid=20849' 2024-08-31T22:20:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=8, entry_tid=20849' 2024-08-31T22:20:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20849 2024-08-31T22:20:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 20849 2024-08-31T22:20:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=75, tag_tid=8, entry_tid=20963], [object_number=74, tag_tid=8, entry_tid=20962], [object_number=73, tag_tid=8, entry_tid=20961], [object_number=72, tag_tid=8, entry_tid=20960]]], state=connected] 2024-08-31T22:20:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=75, tag_tid=8, entry_tid=20963], [object_number=74, tag_tid=8, entry_tid=20962], [object_number=73, tag_tid=8, entry_tid=20961], [object_number=72, tag_tid=8, entry_tid=20960]]], state=connected] 2024-08-31T22:20:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=75, tag_tid=8, entry_tid=20963' 2024-08-31T22:20:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=75, tag_tid=8, entry_tid=20963' 2024-08-31T22:20:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=75, tag_tid=8, entry_tid=20963' '!=' 'object_number=73, tag_tid=8, entry_tid=20849' 2024-08-31T22:20:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=75, tag_tid=8, entry_tid=20963' = 'object_number=73, tag_tid=8, entry_tid=20849' 2024-08-31T22:20:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=75, tag_tid=8, entry_tid=20963' 2024-08-31T22:20:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=75, tag_tid=8, entry_tid=20963' 2024-08-31T22:20:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=75, tag_tid=8, entry_tid=20963' 2024-08-31T22:20:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20963 2024-08-31T22:20:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 20963 2024-08-31T22:20:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=73, tag_tid=8, entry_tid=21065], [object_number=72, tag_tid=8, entry_tid=21064], [object_number=75, tag_tid=8, entry_tid=21063], [object_number=74, tag_tid=8, entry_tid=21062]]], state=connected] 2024-08-31T22:20:13.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:13.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:13.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=73, tag_tid=8, entry_tid=21065], [object_number=72, tag_tid=8, entry_tid=21064], [object_number=75, tag_tid=8, entry_tid=21063], [object_number=74, tag_tid=8, entry_tid=21062]]], state=connected] 2024-08-31T22:20:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=73, tag_tid=8, entry_tid=21065' 2024-08-31T22:20:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=73, tag_tid=8, entry_tid=21065' 2024-08-31T22:20:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=73, tag_tid=8, entry_tid=21065' '!=' 'object_number=75, tag_tid=8, entry_tid=20963' 2024-08-31T22:20:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=73, tag_tid=8, entry_tid=21065' = 'object_number=75, tag_tid=8, entry_tid=20963' 2024-08-31T22:20:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=73, tag_tid=8, entry_tid=21065' 2024-08-31T22:20:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=8, entry_tid=21065' 2024-08-31T22:20:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=8, entry_tid=21065' 2024-08-31T22:20:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21065 2024-08-31T22:20:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 21065 2024-08-31T22:20:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:13.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:13.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:13.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:13.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=75, tag_tid=8, entry_tid=21163], [object_number=74, tag_tid=8, entry_tid=21162], [object_number=73, tag_tid=8, entry_tid=21161], [object_number=72, tag_tid=8, entry_tid=21160]]], state=connected] 2024-08-31T22:20:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=75, tag_tid=8, entry_tid=21163], [object_number=74, tag_tid=8, entry_tid=21162], [object_number=73, tag_tid=8, entry_tid=21161], [object_number=72, tag_tid=8, entry_tid=21160]]], state=connected] 2024-08-31T22:20:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=75, tag_tid=8, entry_tid=21163' 2024-08-31T22:20:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=75, tag_tid=8, entry_tid=21163' 2024-08-31T22:20:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=75, tag_tid=8, entry_tid=21163' '!=' 'object_number=73, tag_tid=8, entry_tid=21065' 2024-08-31T22:20:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=75, tag_tid=8, entry_tid=21163' = 'object_number=73, tag_tid=8, entry_tid=21065' 2024-08-31T22:20:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=75, tag_tid=8, entry_tid=21163' 2024-08-31T22:20:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=75, tag_tid=8, entry_tid=21163' 2024-08-31T22:20:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=75, tag_tid=8, entry_tid=21163' 2024-08-31T22:20:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21163 2024-08-31T22:20:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 21163 2024-08-31T22:20:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=72, tag_tid=8, entry_tid=21268], [object_number=75, tag_tid=8, entry_tid=21267], [object_number=74, tag_tid=8, entry_tid=21266], [object_number=73, tag_tid=8, entry_tid=21265]]], state=connected] 2024-08-31T22:20:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=72, tag_tid=8, entry_tid=21268], [object_number=75, tag_tid=8, entry_tid=21267], [object_number=74, tag_tid=8, entry_tid=21266], [object_number=73, tag_tid=8, entry_tid=21265]]], state=connected] 2024-08-31T22:20:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=72, tag_tid=8, entry_tid=21268' 2024-08-31T22:20:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=72, tag_tid=8, entry_tid=21268' 2024-08-31T22:20:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=72, tag_tid=8, entry_tid=21268' '!=' 'object_number=75, tag_tid=8, entry_tid=21163' 2024-08-31T22:20:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=72, tag_tid=8, entry_tid=21268' = 'object_number=75, tag_tid=8, entry_tid=21163' 2024-08-31T22:20:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=72, tag_tid=8, entry_tid=21268' 2024-08-31T22:20:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=72, tag_tid=8, entry_tid=21268' 2024-08-31T22:20:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=72, tag_tid=8, entry_tid=21268' 2024-08-31T22:20:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21268 2024-08-31T22:20:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 21268 2024-08-31T22:20:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=72, tag_tid=8, entry_tid=21376], [object_number=75, tag_tid=8, entry_tid=21375], [object_number=74, tag_tid=8, entry_tid=21374], [object_number=73, tag_tid=8, entry_tid=21373]]], state=connected] 2024-08-31T22:20:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=72, tag_tid=8, entry_tid=21376], [object_number=75, tag_tid=8, entry_tid=21375], [object_number=74, tag_tid=8, entry_tid=21374], [object_number=73, tag_tid=8, entry_tid=21373]]], state=connected] 2024-08-31T22:20:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=72, tag_tid=8, entry_tid=21376' 2024-08-31T22:20:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=72, tag_tid=8, entry_tid=21376' 2024-08-31T22:20:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=72, tag_tid=8, entry_tid=21376' '!=' 'object_number=72, tag_tid=8, entry_tid=21268' 2024-08-31T22:20:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=72, tag_tid=8, entry_tid=21376' = 'object_number=72, tag_tid=8, entry_tid=21268' 2024-08-31T22:20:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=72, tag_tid=8, entry_tid=21376' 2024-08-31T22:20:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:15.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:15.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=72, tag_tid=8, entry_tid=21376' 2024-08-31T22:20:15.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:15.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:15.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:15.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:15.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:15.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=72, tag_tid=8, entry_tid=21376' 2024-08-31T22:20:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21376 2024-08-31T22:20:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 21376 2024-08-31T22:20:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:15.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:15.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:15.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:15.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=73, tag_tid=8, entry_tid=21481], [object_number=72, tag_tid=8, entry_tid=21480], [object_number=75, tag_tid=8, entry_tid=21479], [object_number=74, tag_tid=8, entry_tid=21478]]], state=connected] 2024-08-31T22:20:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=73, tag_tid=8, entry_tid=21481], [object_number=72, tag_tid=8, entry_tid=21480], [object_number=75, tag_tid=8, entry_tid=21479], [object_number=74, tag_tid=8, entry_tid=21478]]], state=connected] 2024-08-31T22:20:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=73, tag_tid=8, entry_tid=21481' 2024-08-31T22:20:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=73, tag_tid=8, entry_tid=21481' 2024-08-31T22:20:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=73, tag_tid=8, entry_tid=21481' '!=' 'object_number=72, tag_tid=8, entry_tid=21376' 2024-08-31T22:20:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=73, tag_tid=8, entry_tid=21481' = 'object_number=72, tag_tid=8, entry_tid=21376' 2024-08-31T22:20:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=73, tag_tid=8, entry_tid=21481' 2024-08-31T22:20:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=8, entry_tid=21481' 2024-08-31T22:20:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=8, entry_tid=21481' 2024-08-31T22:20:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21481 2024-08-31T22:20:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 21481 2024-08-31T22:20:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:15.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:15.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:15.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:15.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:15.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:15.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:15.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:15.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:15.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:15.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:15.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:15.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:15.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:15.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:15.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:15.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:15.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=72, tag_tid=8, entry_tid=21580], [object_number=75, tag_tid=8, entry_tid=21579], [object_number=74, tag_tid=8, entry_tid=21578], [object_number=73, tag_tid=8, entry_tid=21577]]], state=connected] 2024-08-31T22:20:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=72, tag_tid=8, entry_tid=21580], [object_number=75, tag_tid=8, entry_tid=21579], [object_number=74, tag_tid=8, entry_tid=21578], [object_number=73, tag_tid=8, entry_tid=21577]]], state=connected] 2024-08-31T22:20:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=72, tag_tid=8, entry_tid=21580' 2024-08-31T22:20:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=72, tag_tid=8, entry_tid=21580' 2024-08-31T22:20:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=72, tag_tid=8, entry_tid=21580' '!=' 'object_number=73, tag_tid=8, entry_tid=21481' 2024-08-31T22:20:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=72, tag_tid=8, entry_tid=21580' = 'object_number=73, tag_tid=8, entry_tid=21481' 2024-08-31T22:20:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=72, tag_tid=8, entry_tid=21580' 2024-08-31T22:20:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=72, tag_tid=8, entry_tid=21580' 2024-08-31T22:20:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=72, tag_tid=8, entry_tid=21580' 2024-08-31T22:20:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21580 2024-08-31T22:20:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 21580 2024-08-31T22:20:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:16.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:16.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:16.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:16.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:16.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=75, tag_tid=8, entry_tid=21675], [object_number=74, tag_tid=8, entry_tid=21674], [object_number=73, tag_tid=8, entry_tid=21673], [object_number=72, tag_tid=8, entry_tid=21672]]], state=connected] 2024-08-31T22:20:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=75, tag_tid=8, entry_tid=21675], [object_number=74, tag_tid=8, entry_tid=21674], [object_number=73, tag_tid=8, entry_tid=21673], [object_number=72, tag_tid=8, entry_tid=21672]]], state=connected] 2024-08-31T22:20:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=75, tag_tid=8, entry_tid=21675' 2024-08-31T22:20:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=75, tag_tid=8, entry_tid=21675' 2024-08-31T22:20:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=75, tag_tid=8, entry_tid=21675' '!=' 'object_number=72, tag_tid=8, entry_tid=21580' 2024-08-31T22:20:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:20:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=75, tag_tid=8, entry_tid=21675' = 'object_number=72, tag_tid=8, entry_tid=21580' 2024-08-31T22:20:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=75, tag_tid=8, entry_tid=21675' 2024-08-31T22:20:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=9 2024-08-31T22:20:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=75, tag_tid=8, entry_tid=21675' 2024-08-31T22:20:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:20:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=8 2024-08-31T22:20:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:20:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=75, tag_tid=8, entry_tid=21675' 2024-08-31T22:20:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:20:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:20:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21675 2024-08-31T22:20:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 9 = 8 -a 1 -le 21675 2024-08-31T22:20:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:20:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:20:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:20:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:20:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:20:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:20:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:20:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:20:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:20:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:20:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:20:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:20:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:20:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:20:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:20:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:20:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:20:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:20:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:20:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:20:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:20:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:20:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:20:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:20:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:20:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:20:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:20:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:20:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:20:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:20:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:20:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:20:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:20:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:20:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:20:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:20:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:20:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:20:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:20:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:20:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:20:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:20:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:20:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:20:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:20:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:20:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:20:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:20:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:20:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:20:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:20:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:20:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:20:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:20:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:20:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:20:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:20:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:20:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 18 2024-08-31T22:20:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 18 2024-08-31T22:20:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:20:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=73, tag_tid=9, entry_tid=1], [object_number=72, tag_tid=9, entry_tid=0], [object_number=75, tag_tid=8, entry_tid=21703], [object_number=74, tag_tid=8, entry_tid=21702]]], state=connected] 2024-08-31T22:20:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:20:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=73, tag_tid=9, entry_tid=1' -a 'object_number=73, tag_tid=9, entry_tid=1' = 'object_number=73, tag_tid=9, entry_tid=1' 2024-08-31T22:20:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-08-31T22:20:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap3 2024-08-31T22:20:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:20:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:20:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:20:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-08-31T22:20:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'test': 2024-08-31T22:20:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:20:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:20:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 5 2024-08-31T22:20:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a87b4c6a1b 2024-08-31T22:20:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a87b4c6a1b 2024-08-31T22:20:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:20:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:20:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:20:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:20:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:14:57 2024 2024-08-31T22:20:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:18:19 2024 2024-08-31T22:20:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:20:02 2024 2024-08-31T22:20:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:20:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a87b4c6a1b 2024-08-31T22:20:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:20:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:20:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:20:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:20:18.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n snap2-clean ']' 2024-08-31T22:20:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap2-clean 2024-08-31T22:20:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap2-clean 2024-08-31T22:20:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:20:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:20:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:20:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:20:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:20:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:20:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:20:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:20:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:20:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:20:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap3 2024-08-31T22:20:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap3 2024-08-31T22:20:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:20:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:20:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:20:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:20:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:20:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:20:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:20:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:20:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:20:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:20:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ clean_snap_name=snap3-clean 2024-08-31T22:20:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-08-31T22:20:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:20:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap3-clean 2024-08-31T22:20:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-08-31T22:20:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:20:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror test 2024-08-31T22:20:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:20:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:22.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:20:22.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=12 2024-08-31T22:20:22.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:20:22.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-08-31T22:20:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 96654336 bytes 2024-08-31T22:20:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 126155776 bytes 2024-08-31T22:20:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 187663360 bytes 2024-08-31T22:20:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 275597312 bytes 2024-08-31T22:20:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:500 IOs, 300422144 bytes 2024-08-31T22:20:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:20:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:20:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:20:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-08-31T22:20:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:20:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:20:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:20:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:20:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:20:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-08-31T22:20:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:20:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:20:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:20:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:20:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:20:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-08-31T22:20:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='test: 2024-08-31T22:20:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: global_id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:20:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+replaying 2024-08-31T22:20:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: replaying, {"bytes_per_second":2898008.77,"entries_behind_primary":0,"entries_per_second":180.37,"non_primary_position":{"entry_tid":1,"object_number":73,"tag_tid":9},"primary_position":{"entry_tid":1,"object_number":73,"tag_tid":9}} 2024-08-31T22:20:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: service: mirror.0 on smithi032 2024-08-31T22:20:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:20:25 2024-08-31T22:20:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: peer_sites: 2024-08-31T22:20:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: name: cluster2 2024-08-31T22:20:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+stopped 2024-08-31T22:20:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: local image is primary 2024-08-31T22:20:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:20:35' 2024-08-31T22:20:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ state: .*up+replaying' 2024-08-31T22:20:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: state: up+replaying 2024-08-31T22:20:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ description: .*' 2024-08-31T22:20:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: description: replaying, {"bytes_per_second":2898008.77,"entries_behind_primary":0,"entries_per_second":180.37,"non_primary_position":{"entry_tid":1,"object_number":73,"tag_tid":9},"primary_position":{"entry_tid":1,"object_number":73,"tag_tid":9}} 2024-08-31T22:20:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:20:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:20:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^up+' 2024-08-31T22:20:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:20:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep 'service: *mirror..* on ' 2024-08-31T22:20:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: service: mirror.0 on smithi032 2024-08-31T22:20:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update 2024-08-31T22:20:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-08-31T22:20:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:20:25' 2024-08-31T22:20:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-08-31 22:20:25' 2024-08-31T22:20:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:20:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:20:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'prev_last_update=2024-08-31 22:20:25' 2024-08-31T22:20:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:20:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:20:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-08-31T22:20:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='OKOKOK142574264mirror.0smithi03219.1.1-244-g3091dfc0trueOK42614267mirror.1smithi03219.1.1-244-g3091dfc0falseOKtestdd2c705e-25cf-4565-981f-d48de48eee84up+replayingreplaying, {"bytes_per_second":2898008.77,"entries_behind_primary":0,"entries_per_second":180.37,"non_primary_position":{"entry_tid":1,"object_number":73,"tag_tid":9},"primary_position":{"entry_tid":1,"object_number":73,"tag_tid":9}}42574264mirror.0smithi0322024-08-31 22:20:25cluster24ca32b00-2164-486e-837e-e93795a16af1up+stoppedlocal image is primary2024-08-31 22:20:35' 2024-08-31T22:20:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update state 2024-08-31T22:20:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-08-31T22:20:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:20:25' 2024-08-31T22:20:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-08-31T22:20:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ state=up+replaying 2024-08-31T22:20:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:20:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep up+replaying 2024-08-31T22:20:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:20:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:20:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap4 2024-08-31T22:20:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap4 2024-08-31T22:20:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:20:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:20:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:20:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap4 2024-08-31T22:20:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:21:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-08-31T22:21:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-08-31T22:21:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:21:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:21:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:21:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror test started 2024-08-31T22:21:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:21:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:21:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:21:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-08-31T22:21:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:21:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:21:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:21:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:21:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:21:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:21:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:21:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-08-31T22:21:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:21:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:21:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:21:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:21:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:21:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:21:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:21:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:21:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:21:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:21:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:21:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:21:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:21:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:21:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:21:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:21:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-08-31T22:21:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:21:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:21:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:21:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:21:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:21:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:21:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:21:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:21:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:21:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:21:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:21:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:21:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:21:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:21:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:21:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:21:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=81, tag_tid=11, entry_tid=7261], [object_number=80, tag_tid=11, entry_tid=7260], [object_number=83, tag_tid=11, entry_tid=7259], [object_number=82, tag_tid=11, entry_tid=7258]]], state=connected] 2024-08-31T22:21:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=81, tag_tid=11, entry_tid=7261], [object_number=80, tag_tid=11, entry_tid=7260], [object_number=83, tag_tid=11, entry_tid=7259], [object_number=82, tag_tid=11, entry_tid=7258]]], state=connected] 2024-08-31T22:21:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=81, tag_tid=11, entry_tid=7261' 2024-08-31T22:21:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=81, tag_tid=11, entry_tid=7261' 2024-08-31T22:21:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=81, tag_tid=11, entry_tid=7261' '!=' '' 2024-08-31T22:21:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=81, tag_tid=11, entry_tid=7261' = '' 2024-08-31T22:21:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=81, tag_tid=11, entry_tid=7261' 2024-08-31T22:21:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:07.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=81, tag_tid=11, entry_tid=7261' 2024-08-31T22:21:07.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:07.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=81, tag_tid=11, entry_tid=7261' 2024-08-31T22:21:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7261 2024-08-31T22:21:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 7261 2024-08-31T22:21:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:08.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:08.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:08.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:08.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:08.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=83, tag_tid=11, entry_tid=7343], [object_number=82, tag_tid=11, entry_tid=7342], [object_number=81, tag_tid=11, entry_tid=7341], [object_number=80, tag_tid=11, entry_tid=7340]]], state=connected] 2024-08-31T22:21:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:08.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:08.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:08.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:08.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:08.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:08.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=83, tag_tid=11, entry_tid=7343], [object_number=82, tag_tid=11, entry_tid=7342], [object_number=81, tag_tid=11, entry_tid=7341], [object_number=80, tag_tid=11, entry_tid=7340]]], state=connected] 2024-08-31T22:21:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=83, tag_tid=11, entry_tid=7343' 2024-08-31T22:21:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=83, tag_tid=11, entry_tid=7343' 2024-08-31T22:21:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=83, tag_tid=11, entry_tid=7343' '!=' 'object_number=81, tag_tid=11, entry_tid=7261' 2024-08-31T22:21:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=83, tag_tid=11, entry_tid=7343' = 'object_number=81, tag_tid=11, entry_tid=7261' 2024-08-31T22:21:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=83, tag_tid=11, entry_tid=7343' 2024-08-31T22:21:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=83, tag_tid=11, entry_tid=7343' 2024-08-31T22:21:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:08.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=83, tag_tid=11, entry_tid=7343' 2024-08-31T22:21:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7343 2024-08-31T22:21:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 7343 2024-08-31T22:21:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=82, tag_tid=11, entry_tid=7478], [object_number=81, tag_tid=11, entry_tid=7477], [object_number=80, tag_tid=11, entry_tid=7476], [object_number=83, tag_tid=11, entry_tid=7475]]], state=connected] 2024-08-31T22:21:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=82, tag_tid=11, entry_tid=7478], [object_number=81, tag_tid=11, entry_tid=7477], [object_number=80, tag_tid=11, entry_tid=7476], [object_number=83, tag_tid=11, entry_tid=7475]]], state=connected] 2024-08-31T22:21:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=82, tag_tid=11, entry_tid=7478' 2024-08-31T22:21:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=82, tag_tid=11, entry_tid=7478' 2024-08-31T22:21:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=82, tag_tid=11, entry_tid=7478' '!=' 'object_number=83, tag_tid=11, entry_tid=7343' 2024-08-31T22:21:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=82, tag_tid=11, entry_tid=7478' = 'object_number=83, tag_tid=11, entry_tid=7343' 2024-08-31T22:21:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=82, tag_tid=11, entry_tid=7478' 2024-08-31T22:21:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=82, tag_tid=11, entry_tid=7478' 2024-08-31T22:21:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=82, tag_tid=11, entry_tid=7478' 2024-08-31T22:21:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7478 2024-08-31T22:21:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 7478 2024-08-31T22:21:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:09.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:09.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:09.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=82, tag_tid=11, entry_tid=7582], [object_number=81, tag_tid=11, entry_tid=7581], [object_number=80, tag_tid=11, entry_tid=7580], [object_number=83, tag_tid=11, entry_tid=7579]]], state=connected] 2024-08-31T22:21:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=82, tag_tid=11, entry_tid=7582], [object_number=81, tag_tid=11, entry_tid=7581], [object_number=80, tag_tid=11, entry_tid=7580], [object_number=83, tag_tid=11, entry_tid=7579]]], state=connected] 2024-08-31T22:21:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=82, tag_tid=11, entry_tid=7582' 2024-08-31T22:21:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=82, tag_tid=11, entry_tid=7582' 2024-08-31T22:21:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=82, tag_tid=11, entry_tid=7582' '!=' 'object_number=82, tag_tid=11, entry_tid=7478' 2024-08-31T22:21:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=82, tag_tid=11, entry_tid=7582' = 'object_number=82, tag_tid=11, entry_tid=7478' 2024-08-31T22:21:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=82, tag_tid=11, entry_tid=7582' 2024-08-31T22:21:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=82, tag_tid=11, entry_tid=7582' 2024-08-31T22:21:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=82, tag_tid=11, entry_tid=7582' 2024-08-31T22:21:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7582 2024-08-31T22:21:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 7582 2024-08-31T22:21:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=80, tag_tid=11, entry_tid=7692], [object_number=83, tag_tid=11, entry_tid=7691], [object_number=82, tag_tid=11, entry_tid=7690], [object_number=81, tag_tid=11, entry_tid=7689]]], state=connected] 2024-08-31T22:21:09.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:09.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:09.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:09.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:09.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:09.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:09.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:09.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:09.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:09.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:09.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:09.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:09.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:09.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=80, tag_tid=11, entry_tid=7692], [object_number=83, tag_tid=11, entry_tid=7691], [object_number=82, tag_tid=11, entry_tid=7690], [object_number=81, tag_tid=11, entry_tid=7689]]], state=connected] 2024-08-31T22:21:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=80, tag_tid=11, entry_tid=7692' 2024-08-31T22:21:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=80, tag_tid=11, entry_tid=7692' 2024-08-31T22:21:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=80, tag_tid=11, entry_tid=7692' '!=' 'object_number=82, tag_tid=11, entry_tid=7582' 2024-08-31T22:21:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=80, tag_tid=11, entry_tid=7692' = 'object_number=82, tag_tid=11, entry_tid=7582' 2024-08-31T22:21:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=80, tag_tid=11, entry_tid=7692' 2024-08-31T22:21:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=80, tag_tid=11, entry_tid=7692' 2024-08-31T22:21:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:09.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:09.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:09.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:09.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=80, tag_tid=11, entry_tid=7692' 2024-08-31T22:21:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7692 2024-08-31T22:21:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 7692 2024-08-31T22:21:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=82, tag_tid=11, entry_tid=7794], [object_number=81, tag_tid=11, entry_tid=7793], [object_number=80, tag_tid=11, entry_tid=7792], [object_number=83, tag_tid=11, entry_tid=7791]]], state=connected] 2024-08-31T22:21:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:10.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:10.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=82, tag_tid=11, entry_tid=7794], [object_number=81, tag_tid=11, entry_tid=7793], [object_number=80, tag_tid=11, entry_tid=7792], [object_number=83, tag_tid=11, entry_tid=7791]]], state=connected] 2024-08-31T22:21:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=82, tag_tid=11, entry_tid=7794' 2024-08-31T22:21:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=82, tag_tid=11, entry_tid=7794' 2024-08-31T22:21:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=82, tag_tid=11, entry_tid=7794' '!=' 'object_number=80, tag_tid=11, entry_tid=7692' 2024-08-31T22:21:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=82, tag_tid=11, entry_tid=7794' = 'object_number=80, tag_tid=11, entry_tid=7692' 2024-08-31T22:21:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=82, tag_tid=11, entry_tid=7794' 2024-08-31T22:21:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=82, tag_tid=11, entry_tid=7794' 2024-08-31T22:21:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=82, tag_tid=11, entry_tid=7794' 2024-08-31T22:21:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7794 2024-08-31T22:21:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 7794 2024-08-31T22:21:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:10.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:10.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:10.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=82, tag_tid=11, entry_tid=7882], [object_number=81, tag_tid=11, entry_tid=7881], [object_number=80, tag_tid=11, entry_tid=7880], [object_number=83, tag_tid=11, entry_tid=7879]]], state=connected] 2024-08-31T22:21:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:10.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=82, tag_tid=11, entry_tid=7882], [object_number=81, tag_tid=11, entry_tid=7881], [object_number=80, tag_tid=11, entry_tid=7880], [object_number=83, tag_tid=11, entry_tid=7879]]], state=connected] 2024-08-31T22:21:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=82, tag_tid=11, entry_tid=7882' 2024-08-31T22:21:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=82, tag_tid=11, entry_tid=7882' 2024-08-31T22:21:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=82, tag_tid=11, entry_tid=7882' '!=' 'object_number=82, tag_tid=11, entry_tid=7794' 2024-08-31T22:21:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=82, tag_tid=11, entry_tid=7882' = 'object_number=82, tag_tid=11, entry_tid=7794' 2024-08-31T22:21:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=82, tag_tid=11, entry_tid=7882' 2024-08-31T22:21:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=82, tag_tid=11, entry_tid=7882' 2024-08-31T22:21:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:10.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=82, tag_tid=11, entry_tid=7882' 2024-08-31T22:21:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7882 2024-08-31T22:21:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 7882 2024-08-31T22:21:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=80, tag_tid=11, entry_tid=7988], [object_number=83, tag_tid=11, entry_tid=7987], [object_number=82, tag_tid=11, entry_tid=7986], [object_number=81, tag_tid=11, entry_tid=7985]]], state=connected] 2024-08-31T22:21:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=80, tag_tid=11, entry_tid=7988], [object_number=83, tag_tid=11, entry_tid=7987], [object_number=82, tag_tid=11, entry_tid=7986], [object_number=81, tag_tid=11, entry_tid=7985]]], state=connected] 2024-08-31T22:21:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=80, tag_tid=11, entry_tid=7988' 2024-08-31T22:21:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=80, tag_tid=11, entry_tid=7988' 2024-08-31T22:21:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=80, tag_tid=11, entry_tid=7988' '!=' 'object_number=82, tag_tid=11, entry_tid=7882' 2024-08-31T22:21:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=80, tag_tid=11, entry_tid=7988' = 'object_number=82, tag_tid=11, entry_tid=7882' 2024-08-31T22:21:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=80, tag_tid=11, entry_tid=7988' 2024-08-31T22:21:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:11.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=80, tag_tid=11, entry_tid=7988' 2024-08-31T22:21:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:11.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:11.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:11.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:11.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=80, tag_tid=11, entry_tid=7988' 2024-08-31T22:21:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7988 2024-08-31T22:21:11.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 7988 2024-08-31T22:21:11.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:11.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:11.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=83, tag_tid=11, entry_tid=8095], [object_number=82, tag_tid=11, entry_tid=8094], [object_number=81, tag_tid=11, entry_tid=8093], [object_number=80, tag_tid=11, entry_tid=8092]]], state=connected] 2024-08-31T22:21:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=83, tag_tid=11, entry_tid=8095], [object_number=82, tag_tid=11, entry_tid=8094], [object_number=81, tag_tid=11, entry_tid=8093], [object_number=80, tag_tid=11, entry_tid=8092]]], state=connected] 2024-08-31T22:21:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=83, tag_tid=11, entry_tid=8095' 2024-08-31T22:21:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=83, tag_tid=11, entry_tid=8095' 2024-08-31T22:21:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=83, tag_tid=11, entry_tid=8095' '!=' 'object_number=80, tag_tid=11, entry_tid=7988' 2024-08-31T22:21:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=83, tag_tid=11, entry_tid=8095' = 'object_number=80, tag_tid=11, entry_tid=7988' 2024-08-31T22:21:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=83, tag_tid=11, entry_tid=8095' 2024-08-31T22:21:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=83, tag_tid=11, entry_tid=8095' 2024-08-31T22:21:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=83, tag_tid=11, entry_tid=8095' 2024-08-31T22:21:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8095 2024-08-31T22:21:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 8095 2024-08-31T22:21:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:12.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:12.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=80, tag_tid=11, entry_tid=8192], [object_number=83, tag_tid=11, entry_tid=8191], [object_number=82, tag_tid=11, entry_tid=8190], [object_number=81, tag_tid=11, entry_tid=8189]]], state=connected] 2024-08-31T22:21:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=80, tag_tid=11, entry_tid=8192], [object_number=83, tag_tid=11, entry_tid=8191], [object_number=82, tag_tid=11, entry_tid=8190], [object_number=81, tag_tid=11, entry_tid=8189]]], state=connected] 2024-08-31T22:21:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:12.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=80, tag_tid=11, entry_tid=8192' 2024-08-31T22:21:12.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=80, tag_tid=11, entry_tid=8192' 2024-08-31T22:21:12.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=80, tag_tid=11, entry_tid=8192' '!=' 'object_number=83, tag_tid=11, entry_tid=8095' 2024-08-31T22:21:12.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:12.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=80, tag_tid=11, entry_tid=8192' = 'object_number=83, tag_tid=11, entry_tid=8095' 2024-08-31T22:21:12.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=80, tag_tid=11, entry_tid=8192' 2024-08-31T22:21:12.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:12.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:12.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=80, tag_tid=11, entry_tid=8192' 2024-08-31T22:21:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:12.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:12.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:12.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=80, tag_tid=11, entry_tid=8192' 2024-08-31T22:21:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8192 2024-08-31T22:21:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 8192 2024-08-31T22:21:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=81, tag_tid=11, entry_tid=8289], [object_number=80, tag_tid=11, entry_tid=8288], [object_number=83, tag_tid=11, entry_tid=8287], [object_number=82, tag_tid=11, entry_tid=8286]]], state=connected] 2024-08-31T22:21:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=81, tag_tid=11, entry_tid=8289], [object_number=80, tag_tid=11, entry_tid=8288], [object_number=83, tag_tid=11, entry_tid=8287], [object_number=82, tag_tid=11, entry_tid=8286]]], state=connected] 2024-08-31T22:21:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=81, tag_tid=11, entry_tid=8289' 2024-08-31T22:21:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=81, tag_tid=11, entry_tid=8289' 2024-08-31T22:21:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=81, tag_tid=11, entry_tid=8289' '!=' 'object_number=80, tag_tid=11, entry_tid=8192' 2024-08-31T22:21:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=81, tag_tid=11, entry_tid=8289' = 'object_number=80, tag_tid=11, entry_tid=8192' 2024-08-31T22:21:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=81, tag_tid=11, entry_tid=8289' 2024-08-31T22:21:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=81, tag_tid=11, entry_tid=8289' 2024-08-31T22:21:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:12.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=81, tag_tid=11, entry_tid=8289' 2024-08-31T22:21:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8289 2024-08-31T22:21:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 8289 2024-08-31T22:21:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=81, tag_tid=11, entry_tid=8385], [object_number=80, tag_tid=11, entry_tid=8384], [object_number=83, tag_tid=11, entry_tid=8383], [object_number=82, tag_tid=11, entry_tid=8382]]], state=connected] 2024-08-31T22:21:13.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:13.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:13.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:13.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=81, tag_tid=11, entry_tid=8385], [object_number=80, tag_tid=11, entry_tid=8384], [object_number=83, tag_tid=11, entry_tid=8383], [object_number=82, tag_tid=11, entry_tid=8382]]], state=connected] 2024-08-31T22:21:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=81, tag_tid=11, entry_tid=8385' 2024-08-31T22:21:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=81, tag_tid=11, entry_tid=8385' 2024-08-31T22:21:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=81, tag_tid=11, entry_tid=8385' '!=' 'object_number=81, tag_tid=11, entry_tid=8289' 2024-08-31T22:21:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=81, tag_tid=11, entry_tid=8385' = 'object_number=81, tag_tid=11, entry_tid=8289' 2024-08-31T22:21:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=81, tag_tid=11, entry_tid=8385' 2024-08-31T22:21:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=81, tag_tid=11, entry_tid=8385' 2024-08-31T22:21:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=81, tag_tid=11, entry_tid=8385' 2024-08-31T22:21:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8385 2024-08-31T22:21:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 8385 2024-08-31T22:21:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:13.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:13.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=81, tag_tid=11, entry_tid=8489], [object_number=80, tag_tid=11, entry_tid=8488], [object_number=83, tag_tid=11, entry_tid=8487], [object_number=82, tag_tid=11, entry_tid=8486]]], state=connected] 2024-08-31T22:21:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=81, tag_tid=11, entry_tid=8489], [object_number=80, tag_tid=11, entry_tid=8488], [object_number=83, tag_tid=11, entry_tid=8487], [object_number=82, tag_tid=11, entry_tid=8486]]], state=connected] 2024-08-31T22:21:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=81, tag_tid=11, entry_tid=8489' 2024-08-31T22:21:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=81, tag_tid=11, entry_tid=8489' 2024-08-31T22:21:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=81, tag_tid=11, entry_tid=8489' '!=' 'object_number=81, tag_tid=11, entry_tid=8385' 2024-08-31T22:21:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=81, tag_tid=11, entry_tid=8489' = 'object_number=81, tag_tid=11, entry_tid=8385' 2024-08-31T22:21:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=81, tag_tid=11, entry_tid=8489' 2024-08-31T22:21:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=81, tag_tid=11, entry_tid=8489' 2024-08-31T22:21:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=81, tag_tid=11, entry_tid=8489' 2024-08-31T22:21:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8489 2024-08-31T22:21:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 8489 2024-08-31T22:21:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=82, tag_tid=11, entry_tid=8602], [object_number=81, tag_tid=11, entry_tid=8601], [object_number=80, tag_tid=11, entry_tid=8600], [object_number=83, tag_tid=11, entry_tid=8599]]], state=connected] 2024-08-31T22:21:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=82, tag_tid=11, entry_tid=8602], [object_number=81, tag_tid=11, entry_tid=8601], [object_number=80, tag_tid=11, entry_tid=8600], [object_number=83, tag_tid=11, entry_tid=8599]]], state=connected] 2024-08-31T22:21:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=82, tag_tid=11, entry_tid=8602' 2024-08-31T22:21:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=82, tag_tid=11, entry_tid=8602' 2024-08-31T22:21:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=82, tag_tid=11, entry_tid=8602' '!=' 'object_number=81, tag_tid=11, entry_tid=8489' 2024-08-31T22:21:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=82, tag_tid=11, entry_tid=8602' = 'object_number=81, tag_tid=11, entry_tid=8489' 2024-08-31T22:21:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=82, tag_tid=11, entry_tid=8602' 2024-08-31T22:21:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=82, tag_tid=11, entry_tid=8602' 2024-08-31T22:21:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=82, tag_tid=11, entry_tid=8602' 2024-08-31T22:21:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8602 2024-08-31T22:21:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 8602 2024-08-31T22:21:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:14.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:14.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:14.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=83, tag_tid=11, entry_tid=8703], [object_number=82, tag_tid=11, entry_tid=8702], [object_number=81, tag_tid=11, entry_tid=8701], [object_number=80, tag_tid=11, entry_tid=8700]]], state=connected] 2024-08-31T22:21:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=83, tag_tid=11, entry_tid=8703], [object_number=82, tag_tid=11, entry_tid=8702], [object_number=81, tag_tid=11, entry_tid=8701], [object_number=80, tag_tid=11, entry_tid=8700]]], state=connected] 2024-08-31T22:21:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=83, tag_tid=11, entry_tid=8703' 2024-08-31T22:21:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=83, tag_tid=11, entry_tid=8703' 2024-08-31T22:21:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=83, tag_tid=11, entry_tid=8703' '!=' 'object_number=82, tag_tid=11, entry_tid=8602' 2024-08-31T22:21:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=83, tag_tid=11, entry_tid=8703' = 'object_number=82, tag_tid=11, entry_tid=8602' 2024-08-31T22:21:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=83, tag_tid=11, entry_tid=8703' 2024-08-31T22:21:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=83, tag_tid=11, entry_tid=8703' 2024-08-31T22:21:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=83, tag_tid=11, entry_tid=8703' 2024-08-31T22:21:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8703 2024-08-31T22:21:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 8703 2024-08-31T22:21:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=81, tag_tid=11, entry_tid=8785], [object_number=80, tag_tid=11, entry_tid=8784], [object_number=83, tag_tid=11, entry_tid=8783], [object_number=82, tag_tid=11, entry_tid=8782]]], state=connected] 2024-08-31T22:21:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=81, tag_tid=11, entry_tid=8785], [object_number=80, tag_tid=11, entry_tid=8784], [object_number=83, tag_tid=11, entry_tid=8783], [object_number=82, tag_tid=11, entry_tid=8782]]], state=connected] 2024-08-31T22:21:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=81, tag_tid=11, entry_tid=8785' 2024-08-31T22:21:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=81, tag_tid=11, entry_tid=8785' 2024-08-31T22:21:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=81, tag_tid=11, entry_tid=8785' '!=' 'object_number=83, tag_tid=11, entry_tid=8703' 2024-08-31T22:21:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=81, tag_tid=11, entry_tid=8785' = 'object_number=83, tag_tid=11, entry_tid=8703' 2024-08-31T22:21:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=81, tag_tid=11, entry_tid=8785' 2024-08-31T22:21:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=81, tag_tid=11, entry_tid=8785' 2024-08-31T22:21:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=81, tag_tid=11, entry_tid=8785' 2024-08-31T22:21:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8785 2024-08-31T22:21:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 8785 2024-08-31T22:21:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:15.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=83, tag_tid=11, entry_tid=8883], [object_number=82, tag_tid=11, entry_tid=8882], [object_number=81, tag_tid=11, entry_tid=8881], [object_number=80, tag_tid=11, entry_tid=8880]]], state=connected] 2024-08-31T22:21:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=83, tag_tid=11, entry_tid=8883], [object_number=82, tag_tid=11, entry_tid=8882], [object_number=81, tag_tid=11, entry_tid=8881], [object_number=80, tag_tid=11, entry_tid=8880]]], state=connected] 2024-08-31T22:21:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=83, tag_tid=11, entry_tid=8883' 2024-08-31T22:21:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=83, tag_tid=11, entry_tid=8883' 2024-08-31T22:21:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=83, tag_tid=11, entry_tid=8883' '!=' 'object_number=81, tag_tid=11, entry_tid=8785' 2024-08-31T22:21:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=83, tag_tid=11, entry_tid=8883' = 'object_number=81, tag_tid=11, entry_tid=8785' 2024-08-31T22:21:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=83, tag_tid=11, entry_tid=8883' 2024-08-31T22:21:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=83, tag_tid=11, entry_tid=8883' 2024-08-31T22:21:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=83, tag_tid=11, entry_tid=8883' 2024-08-31T22:21:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8883 2024-08-31T22:21:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 8883 2024-08-31T22:21:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=81, tag_tid=11, entry_tid=8973], [object_number=80, tag_tid=11, entry_tid=8972], [object_number=83, tag_tid=11, entry_tid=8971], [object_number=82, tag_tid=11, entry_tid=8970]]], state=connected] 2024-08-31T22:21:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:16.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=81, tag_tid=11, entry_tid=8973], [object_number=80, tag_tid=11, entry_tid=8972], [object_number=83, tag_tid=11, entry_tid=8971], [object_number=82, tag_tid=11, entry_tid=8970]]], state=connected] 2024-08-31T22:21:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=81, tag_tid=11, entry_tid=8973' 2024-08-31T22:21:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=81, tag_tid=11, entry_tid=8973' 2024-08-31T22:21:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=81, tag_tid=11, entry_tid=8973' '!=' 'object_number=83, tag_tid=11, entry_tid=8883' 2024-08-31T22:21:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=81, tag_tid=11, entry_tid=8973' = 'object_number=83, tag_tid=11, entry_tid=8883' 2024-08-31T22:21:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=81, tag_tid=11, entry_tid=8973' 2024-08-31T22:21:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=81, tag_tid=11, entry_tid=8973' 2024-08-31T22:21:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=81, tag_tid=11, entry_tid=8973' 2024-08-31T22:21:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8973 2024-08-31T22:21:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 8973 2024-08-31T22:21:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=83, tag_tid=11, entry_tid=9067], [object_number=82, tag_tid=11, entry_tid=9066], [object_number=81, tag_tid=11, entry_tid=9065], [object_number=80, tag_tid=11, entry_tid=9064]]], state=connected] 2024-08-31T22:21:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:16.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:16.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:16.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:16.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:16.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=83, tag_tid=11, entry_tid=9067], [object_number=82, tag_tid=11, entry_tid=9066], [object_number=81, tag_tid=11, entry_tid=9065], [object_number=80, tag_tid=11, entry_tid=9064]]], state=connected] 2024-08-31T22:21:16.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=83, tag_tid=11, entry_tid=9067' 2024-08-31T22:21:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=83, tag_tid=11, entry_tid=9067' 2024-08-31T22:21:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=83, tag_tid=11, entry_tid=9067' '!=' 'object_number=81, tag_tid=11, entry_tid=8973' 2024-08-31T22:21:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=83, tag_tid=11, entry_tid=9067' = 'object_number=81, tag_tid=11, entry_tid=8973' 2024-08-31T22:21:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=83, tag_tid=11, entry_tid=9067' 2024-08-31T22:21:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=83, tag_tid=11, entry_tid=9067' 2024-08-31T22:21:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=83, tag_tid=11, entry_tid=9067' 2024-08-31T22:21:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9067 2024-08-31T22:21:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 9067 2024-08-31T22:21:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=83, tag_tid=11, entry_tid=9131], [object_number=86, tag_tid=11, entry_tid=9130], [object_number=81, tag_tid=11, entry_tid=9129], [object_number=80, tag_tid=11, entry_tid=9128]]], state=connected] 2024-08-31T22:21:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 20 2024-08-31T22:21:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=83, tag_tid=11, entry_tid=9131], [object_number=86, tag_tid=11, entry_tid=9130], [object_number=81, tag_tid=11, entry_tid=9129], [object_number=80, tag_tid=11, entry_tid=9128]]], state=connected] 2024-08-31T22:21:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=83, tag_tid=11, entry_tid=9131' 2024-08-31T22:21:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=83, tag_tid=11, entry_tid=9131' 2024-08-31T22:21:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=83, tag_tid=11, entry_tid=9131' '!=' 'object_number=83, tag_tid=11, entry_tid=9067' 2024-08-31T22:21:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=83, tag_tid=11, entry_tid=9131' = 'object_number=83, tag_tid=11, entry_tid=9067' 2024-08-31T22:21:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=83, tag_tid=11, entry_tid=9131' 2024-08-31T22:21:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=83, tag_tid=11, entry_tid=9131' 2024-08-31T22:21:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=83, tag_tid=11, entry_tid=9131' 2024-08-31T22:21:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9131 2024-08-31T22:21:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 9131 2024-08-31T22:21:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=9174], [object_number=85, tag_tid=11, entry_tid=9173], [object_number=84, tag_tid=11, entry_tid=9172], [object_number=87, tag_tid=11, entry_tid=9171]]], state=connected] 2024-08-31T22:21:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:17.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=9174], [object_number=85, tag_tid=11, entry_tid=9173], [object_number=84, tag_tid=11, entry_tid=9172], [object_number=87, tag_tid=11, entry_tid=9171]]], state=connected] 2024-08-31T22:21:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=86, tag_tid=11, entry_tid=9174' 2024-08-31T22:21:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=86, tag_tid=11, entry_tid=9174' 2024-08-31T22:21:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=9174' '!=' 'object_number=83, tag_tid=11, entry_tid=9131' 2024-08-31T22:21:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=9174' = 'object_number=83, tag_tid=11, entry_tid=9131' 2024-08-31T22:21:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=86, tag_tid=11, entry_tid=9174' 2024-08-31T22:21:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=9174' 2024-08-31T22:21:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=9174' 2024-08-31T22:21:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9174 2024-08-31T22:21:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 9174 2024-08-31T22:21:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:18.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:18.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:18.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:18.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:18.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:18.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:18.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:18.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:18.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:18.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:18.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:18.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=85, tag_tid=11, entry_tid=9281], [object_number=84, tag_tid=11, entry_tid=9280], [object_number=87, tag_tid=11, entry_tid=9279], [object_number=86, tag_tid=11, entry_tid=9278]]], state=connected] 2024-08-31T22:21:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:18.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:18.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:18.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=85, tag_tid=11, entry_tid=9281], [object_number=84, tag_tid=11, entry_tid=9280], [object_number=87, tag_tid=11, entry_tid=9279], [object_number=86, tag_tid=11, entry_tid=9278]]], state=connected] 2024-08-31T22:21:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=85, tag_tid=11, entry_tid=9281' 2024-08-31T22:21:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=85, tag_tid=11, entry_tid=9281' 2024-08-31T22:21:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=85, tag_tid=11, entry_tid=9281' '!=' 'object_number=86, tag_tid=11, entry_tid=9174' 2024-08-31T22:21:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=85, tag_tid=11, entry_tid=9281' = 'object_number=86, tag_tid=11, entry_tid=9174' 2024-08-31T22:21:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=85, tag_tid=11, entry_tid=9281' 2024-08-31T22:21:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:18.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=85, tag_tid=11, entry_tid=9281' 2024-08-31T22:21:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:18.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:18.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=85, tag_tid=11, entry_tid=9281' 2024-08-31T22:21:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9281 2024-08-31T22:21:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 9281 2024-08-31T22:21:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:18.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:18.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:18.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=9384], [object_number=87, tag_tid=11, entry_tid=9383], [object_number=86, tag_tid=11, entry_tid=9382], [object_number=85, tag_tid=11, entry_tid=9381]]], state=connected] 2024-08-31T22:21:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=9384], [object_number=87, tag_tid=11, entry_tid=9383], [object_number=86, tag_tid=11, entry_tid=9382], [object_number=85, tag_tid=11, entry_tid=9381]]], state=connected] 2024-08-31T22:21:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=84, tag_tid=11, entry_tid=9384' 2024-08-31T22:21:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=84, tag_tid=11, entry_tid=9384' 2024-08-31T22:21:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=9384' '!=' 'object_number=85, tag_tid=11, entry_tid=9281' 2024-08-31T22:21:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=9384' = 'object_number=85, tag_tid=11, entry_tid=9281' 2024-08-31T22:21:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=84, tag_tid=11, entry_tid=9384' 2024-08-31T22:21:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=9384' 2024-08-31T22:21:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=9384' 2024-08-31T22:21:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9384 2024-08-31T22:21:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 9384 2024-08-31T22:21:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:19.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:19.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:19.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:19.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:19.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:19.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:19.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:19.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:19.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:19.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:19.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:19.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:19.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:19.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:19.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:19.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:19.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:19.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:19.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:19.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:19.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=87, tag_tid=11, entry_tid=9487], [object_number=86, tag_tid=11, entry_tid=9486], [object_number=85, tag_tid=11, entry_tid=9485], [object_number=84, tag_tid=11, entry_tid=9484]]], state=connected] 2024-08-31T22:21:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=87, tag_tid=11, entry_tid=9487], [object_number=86, tag_tid=11, entry_tid=9486], [object_number=85, tag_tid=11, entry_tid=9485], [object_number=84, tag_tid=11, entry_tid=9484]]], state=connected] 2024-08-31T22:21:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=87, tag_tid=11, entry_tid=9487' 2024-08-31T22:21:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=87, tag_tid=11, entry_tid=9487' 2024-08-31T22:21:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=87, tag_tid=11, entry_tid=9487' '!=' 'object_number=84, tag_tid=11, entry_tid=9384' 2024-08-31T22:21:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=87, tag_tid=11, entry_tid=9487' = 'object_number=84, tag_tid=11, entry_tid=9384' 2024-08-31T22:21:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=87, tag_tid=11, entry_tid=9487' 2024-08-31T22:21:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=87, tag_tid=11, entry_tid=9487' 2024-08-31T22:21:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=87, tag_tid=11, entry_tid=9487' 2024-08-31T22:21:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9487 2024-08-31T22:21:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 9487 2024-08-31T22:21:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:19.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:19.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:19.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:19.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:19.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:19.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=85, tag_tid=11, entry_tid=9577], [object_number=84, tag_tid=11, entry_tid=9576], [object_number=87, tag_tid=11, entry_tid=9575], [object_number=86, tag_tid=11, entry_tid=9574]]], state=connected] 2024-08-31T22:21:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=85, tag_tid=11, entry_tid=9577], [object_number=84, tag_tid=11, entry_tid=9576], [object_number=87, tag_tid=11, entry_tid=9575], [object_number=86, tag_tid=11, entry_tid=9574]]], state=connected] 2024-08-31T22:21:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=85, tag_tid=11, entry_tid=9577' 2024-08-31T22:21:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=85, tag_tid=11, entry_tid=9577' 2024-08-31T22:21:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=85, tag_tid=11, entry_tid=9577' '!=' 'object_number=87, tag_tid=11, entry_tid=9487' 2024-08-31T22:21:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=85, tag_tid=11, entry_tid=9577' = 'object_number=87, tag_tid=11, entry_tid=9487' 2024-08-31T22:21:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=85, tag_tid=11, entry_tid=9577' 2024-08-31T22:21:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:19.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=85, tag_tid=11, entry_tid=9577' 2024-08-31T22:21:19.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:19.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:19.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=85, tag_tid=11, entry_tid=9577' 2024-08-31T22:21:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:19.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9577 2024-08-31T22:21:19.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 9577 2024-08-31T22:21:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=87, tag_tid=11, entry_tid=9663], [object_number=86, tag_tid=11, entry_tid=9662], [object_number=85, tag_tid=11, entry_tid=9661], [object_number=84, tag_tid=11, entry_tid=9660]]], state=connected] 2024-08-31T22:21:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:20.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:20.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:20.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=87, tag_tid=11, entry_tid=9663], [object_number=86, tag_tid=11, entry_tid=9662], [object_number=85, tag_tid=11, entry_tid=9661], [object_number=84, tag_tid=11, entry_tid=9660]]], state=connected] 2024-08-31T22:21:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=87, tag_tid=11, entry_tid=9663' 2024-08-31T22:21:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=87, tag_tid=11, entry_tid=9663' 2024-08-31T22:21:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=87, tag_tid=11, entry_tid=9663' '!=' 'object_number=85, tag_tid=11, entry_tid=9577' 2024-08-31T22:21:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=87, tag_tid=11, entry_tid=9663' = 'object_number=85, tag_tid=11, entry_tid=9577' 2024-08-31T22:21:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=87, tag_tid=11, entry_tid=9663' 2024-08-31T22:21:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=87, tag_tid=11, entry_tid=9663' 2024-08-31T22:21:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=87, tag_tid=11, entry_tid=9663' 2024-08-31T22:21:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9663 2024-08-31T22:21:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 9663 2024-08-31T22:21:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:20.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:20.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=9766], [object_number=85, tag_tid=11, entry_tid=9765], [object_number=84, tag_tid=11, entry_tid=9764], [object_number=87, tag_tid=11, entry_tid=9763]]], state=connected] 2024-08-31T22:21:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=9766], [object_number=85, tag_tid=11, entry_tid=9765], [object_number=84, tag_tid=11, entry_tid=9764], [object_number=87, tag_tid=11, entry_tid=9763]]], state=connected] 2024-08-31T22:21:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=86, tag_tid=11, entry_tid=9766' 2024-08-31T22:21:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=86, tag_tid=11, entry_tid=9766' 2024-08-31T22:21:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=9766' '!=' 'object_number=87, tag_tid=11, entry_tid=9663' 2024-08-31T22:21:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=9766' = 'object_number=87, tag_tid=11, entry_tid=9663' 2024-08-31T22:21:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=86, tag_tid=11, entry_tid=9766' 2024-08-31T22:21:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=9766' 2024-08-31T22:21:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:20.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=9766' 2024-08-31T22:21:20.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:20.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9766 2024-08-31T22:21:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 9766 2024-08-31T22:21:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=9878], [object_number=85, tag_tid=11, entry_tid=9877], [object_number=84, tag_tid=11, entry_tid=9876], [object_number=87, tag_tid=11, entry_tid=9875]]], state=connected] 2024-08-31T22:21:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:21.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=9878], [object_number=85, tag_tid=11, entry_tid=9877], [object_number=84, tag_tid=11, entry_tid=9876], [object_number=87, tag_tid=11, entry_tid=9875]]], state=connected] 2024-08-31T22:21:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=86, tag_tid=11, entry_tid=9878' 2024-08-31T22:21:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=86, tag_tid=11, entry_tid=9878' 2024-08-31T22:21:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=9878' '!=' 'object_number=86, tag_tid=11, entry_tid=9766' 2024-08-31T22:21:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=9878' = 'object_number=86, tag_tid=11, entry_tid=9766' 2024-08-31T22:21:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=86, tag_tid=11, entry_tid=9878' 2024-08-31T22:21:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=9878' 2024-08-31T22:21:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=9878' 2024-08-31T22:21:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9878 2024-08-31T22:21:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 9878 2024-08-31T22:21:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=9978], [object_number=85, tag_tid=11, entry_tid=9977], [object_number=84, tag_tid=11, entry_tid=9976], [object_number=87, tag_tid=11, entry_tid=9975]]], state=connected] 2024-08-31T22:21:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=9978], [object_number=85, tag_tid=11, entry_tid=9977], [object_number=84, tag_tid=11, entry_tid=9976], [object_number=87, tag_tid=11, entry_tid=9975]]], state=connected] 2024-08-31T22:21:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=86, tag_tid=11, entry_tid=9978' 2024-08-31T22:21:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=86, tag_tid=11, entry_tid=9978' 2024-08-31T22:21:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=9978' '!=' 'object_number=86, tag_tid=11, entry_tid=9878' 2024-08-31T22:21:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=9978' = 'object_number=86, tag_tid=11, entry_tid=9878' 2024-08-31T22:21:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=86, tag_tid=11, entry_tid=9978' 2024-08-31T22:21:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=9978' 2024-08-31T22:21:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=9978' 2024-08-31T22:21:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9978 2024-08-31T22:21:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 9978 2024-08-31T22:21:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:21.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:21.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:21.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:21.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=10066], [object_number=85, tag_tid=11, entry_tid=10065], [object_number=84, tag_tid=11, entry_tid=10064], [object_number=87, tag_tid=11, entry_tid=10063]]], state=connected] 2024-08-31T22:21:22.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:22.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=10066], [object_number=85, tag_tid=11, entry_tid=10065], [object_number=84, tag_tid=11, entry_tid=10064], [object_number=87, tag_tid=11, entry_tid=10063]]], state=connected] 2024-08-31T22:21:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=86, tag_tid=11, entry_tid=10066' 2024-08-31T22:21:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=86, tag_tid=11, entry_tid=10066' 2024-08-31T22:21:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=10066' '!=' 'object_number=86, tag_tid=11, entry_tid=9978' 2024-08-31T22:21:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=10066' = 'object_number=86, tag_tid=11, entry_tid=9978' 2024-08-31T22:21:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=86, tag_tid=11, entry_tid=10066' 2024-08-31T22:21:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:22.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:22.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=10066' 2024-08-31T22:21:22.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:22.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:22.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:22.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:22.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=10066' 2024-08-31T22:21:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10066 2024-08-31T22:21:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 10066 2024-08-31T22:21:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=10162], [object_number=85, tag_tid=11, entry_tid=10161], [object_number=84, tag_tid=11, entry_tid=10160], [object_number=87, tag_tid=11, entry_tid=10159]]], state=connected] 2024-08-31T22:21:22.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=10162], [object_number=85, tag_tid=11, entry_tid=10161], [object_number=84, tag_tid=11, entry_tid=10160], [object_number=87, tag_tid=11, entry_tid=10159]]], state=connected] 2024-08-31T22:21:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=86, tag_tid=11, entry_tid=10162' 2024-08-31T22:21:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=86, tag_tid=11, entry_tid=10162' 2024-08-31T22:21:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=10162' '!=' 'object_number=86, tag_tid=11, entry_tid=10066' 2024-08-31T22:21:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=10162' = 'object_number=86, tag_tid=11, entry_tid=10066' 2024-08-31T22:21:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=86, tag_tid=11, entry_tid=10162' 2024-08-31T22:21:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=10162' 2024-08-31T22:21:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:22.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:22.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=10162' 2024-08-31T22:21:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10162 2024-08-31T22:21:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 10162 2024-08-31T22:21:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:22.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=10258], [object_number=85, tag_tid=11, entry_tid=10257], [object_number=84, tag_tid=11, entry_tid=10256], [object_number=87, tag_tid=11, entry_tid=10255]]], state=connected] 2024-08-31T22:21:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=10258], [object_number=85, tag_tid=11, entry_tid=10257], [object_number=84, tag_tid=11, entry_tid=10256], [object_number=87, tag_tid=11, entry_tid=10255]]], state=connected] 2024-08-31T22:21:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=86, tag_tid=11, entry_tid=10258' 2024-08-31T22:21:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=86, tag_tid=11, entry_tid=10258' 2024-08-31T22:21:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=10258' '!=' 'object_number=86, tag_tid=11, entry_tid=10162' 2024-08-31T22:21:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=10258' = 'object_number=86, tag_tid=11, entry_tid=10162' 2024-08-31T22:21:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=86, tag_tid=11, entry_tid=10258' 2024-08-31T22:21:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=10258' 2024-08-31T22:21:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=10258' 2024-08-31T22:21:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10258 2024-08-31T22:21:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 10258 2024-08-31T22:21:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=10352], [object_number=87, tag_tid=11, entry_tid=10351], [object_number=86, tag_tid=11, entry_tid=10350], [object_number=85, tag_tid=11, entry_tid=10349]]], state=connected] 2024-08-31T22:21:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:23.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:23.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:23.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:23.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:23.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:23.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:23.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=10352], [object_number=87, tag_tid=11, entry_tid=10351], [object_number=86, tag_tid=11, entry_tid=10350], [object_number=85, tag_tid=11, entry_tid=10349]]], state=connected] 2024-08-31T22:21:23.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=84, tag_tid=11, entry_tid=10352' 2024-08-31T22:21:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=84, tag_tid=11, entry_tid=10352' 2024-08-31T22:21:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=10352' '!=' 'object_number=86, tag_tid=11, entry_tid=10258' 2024-08-31T22:21:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=10352' = 'object_number=86, tag_tid=11, entry_tid=10258' 2024-08-31T22:21:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=84, tag_tid=11, entry_tid=10352' 2024-08-31T22:21:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=10352' 2024-08-31T22:21:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=10352' 2024-08-31T22:21:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10352 2024-08-31T22:21:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 10352 2024-08-31T22:21:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=10444], [object_number=87, tag_tid=11, entry_tid=10443], [object_number=86, tag_tid=11, entry_tid=10442], [object_number=85, tag_tid=11, entry_tid=10441]]], state=connected] 2024-08-31T22:21:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:24.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:24.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:24.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:24.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:24.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:24.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:24.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:24.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:24.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:24.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:24.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:24.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:24.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=10444], [object_number=87, tag_tid=11, entry_tid=10443], [object_number=86, tag_tid=11, entry_tid=10442], [object_number=85, tag_tid=11, entry_tid=10441]]], state=connected] 2024-08-31T22:21:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=84, tag_tid=11, entry_tid=10444' 2024-08-31T22:21:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=84, tag_tid=11, entry_tid=10444' 2024-08-31T22:21:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=10444' '!=' 'object_number=84, tag_tid=11, entry_tid=10352' 2024-08-31T22:21:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=10444' = 'object_number=84, tag_tid=11, entry_tid=10352' 2024-08-31T22:21:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=84, tag_tid=11, entry_tid=10444' 2024-08-31T22:21:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=10444' 2024-08-31T22:21:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=10444' 2024-08-31T22:21:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10444 2024-08-31T22:21:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 10444 2024-08-31T22:21:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=10532], [object_number=87, tag_tid=11, entry_tid=10531], [object_number=86, tag_tid=11, entry_tid=10530], [object_number=85, tag_tid=11, entry_tid=10529]]], state=connected] 2024-08-31T22:21:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=10532], [object_number=87, tag_tid=11, entry_tid=10531], [object_number=86, tag_tid=11, entry_tid=10530], [object_number=85, tag_tid=11, entry_tid=10529]]], state=connected] 2024-08-31T22:21:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=84, tag_tid=11, entry_tid=10532' 2024-08-31T22:21:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=84, tag_tid=11, entry_tid=10532' 2024-08-31T22:21:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=10532' '!=' 'object_number=84, tag_tid=11, entry_tid=10444' 2024-08-31T22:21:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=10532' = 'object_number=84, tag_tid=11, entry_tid=10444' 2024-08-31T22:21:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=84, tag_tid=11, entry_tid=10532' 2024-08-31T22:21:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=10532' 2024-08-31T22:21:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=10532' 2024-08-31T22:21:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10532 2024-08-31T22:21:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 10532 2024-08-31T22:21:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=87, tag_tid=11, entry_tid=10623], [object_number=86, tag_tid=11, entry_tid=10622], [object_number=85, tag_tid=11, entry_tid=10621], [object_number=84, tag_tid=11, entry_tid=10620]]], state=connected] 2024-08-31T22:21:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:25.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=87, tag_tid=11, entry_tid=10623], [object_number=86, tag_tid=11, entry_tid=10622], [object_number=85, tag_tid=11, entry_tid=10621], [object_number=84, tag_tid=11, entry_tid=10620]]], state=connected] 2024-08-31T22:21:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=87, tag_tid=11, entry_tid=10623' 2024-08-31T22:21:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=87, tag_tid=11, entry_tid=10623' 2024-08-31T22:21:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=87, tag_tid=11, entry_tid=10623' '!=' 'object_number=84, tag_tid=11, entry_tid=10532' 2024-08-31T22:21:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=87, tag_tid=11, entry_tid=10623' = 'object_number=84, tag_tid=11, entry_tid=10532' 2024-08-31T22:21:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=87, tag_tid=11, entry_tid=10623' 2024-08-31T22:21:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=87, tag_tid=11, entry_tid=10623' 2024-08-31T22:21:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=87, tag_tid=11, entry_tid=10623' 2024-08-31T22:21:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10623 2024-08-31T22:21:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 10623 2024-08-31T22:21:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=10706], [object_number=85, tag_tid=11, entry_tid=10705], [object_number=84, tag_tid=11, entry_tid=10704], [object_number=87, tag_tid=11, entry_tid=10703]]], state=connected] 2024-08-31T22:21:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=10706], [object_number=85, tag_tid=11, entry_tid=10705], [object_number=84, tag_tid=11, entry_tid=10704], [object_number=87, tag_tid=11, entry_tid=10703]]], state=connected] 2024-08-31T22:21:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=86, tag_tid=11, entry_tid=10706' 2024-08-31T22:21:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=86, tag_tid=11, entry_tid=10706' 2024-08-31T22:21:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=10706' '!=' 'object_number=87, tag_tid=11, entry_tid=10623' 2024-08-31T22:21:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=10706' = 'object_number=87, tag_tid=11, entry_tid=10623' 2024-08-31T22:21:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=86, tag_tid=11, entry_tid=10706' 2024-08-31T22:21:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=10706' 2024-08-31T22:21:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=10706' 2024-08-31T22:21:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10706 2024-08-31T22:21:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 10706 2024-08-31T22:21:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=85, tag_tid=11, entry_tid=10813], [object_number=84, tag_tid=11, entry_tid=10812], [object_number=87, tag_tid=11, entry_tid=10811], [object_number=86, tag_tid=11, entry_tid=10810]]], state=connected] 2024-08-31T22:21:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=85, tag_tid=11, entry_tid=10813], [object_number=84, tag_tid=11, entry_tid=10812], [object_number=87, tag_tid=11, entry_tid=10811], [object_number=86, tag_tid=11, entry_tid=10810]]], state=connected] 2024-08-31T22:21:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=85, tag_tid=11, entry_tid=10813' 2024-08-31T22:21:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=85, tag_tid=11, entry_tid=10813' 2024-08-31T22:21:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=85, tag_tid=11, entry_tid=10813' '!=' 'object_number=86, tag_tid=11, entry_tid=10706' 2024-08-31T22:21:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=85, tag_tid=11, entry_tid=10813' = 'object_number=86, tag_tid=11, entry_tid=10706' 2024-08-31T22:21:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=85, tag_tid=11, entry_tid=10813' 2024-08-31T22:21:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=85, tag_tid=11, entry_tid=10813' 2024-08-31T22:21:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:26.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=85, tag_tid=11, entry_tid=10813' 2024-08-31T22:21:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10813 2024-08-31T22:21:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 10813 2024-08-31T22:21:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=85, tag_tid=11, entry_tid=10917], [object_number=84, tag_tid=11, entry_tid=10916], [object_number=87, tag_tid=11, entry_tid=10915], [object_number=86, tag_tid=11, entry_tid=10914]]], state=connected] 2024-08-31T22:21:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=85, tag_tid=11, entry_tid=10917], [object_number=84, tag_tid=11, entry_tid=10916], [object_number=87, tag_tid=11, entry_tid=10915], [object_number=86, tag_tid=11, entry_tid=10914]]], state=connected] 2024-08-31T22:21:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=85, tag_tid=11, entry_tid=10917' 2024-08-31T22:21:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=85, tag_tid=11, entry_tid=10917' 2024-08-31T22:21:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=85, tag_tid=11, entry_tid=10917' '!=' 'object_number=85, tag_tid=11, entry_tid=10813' 2024-08-31T22:21:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=85, tag_tid=11, entry_tid=10917' = 'object_number=85, tag_tid=11, entry_tid=10813' 2024-08-31T22:21:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=85, tag_tid=11, entry_tid=10917' 2024-08-31T22:21:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=85, tag_tid=11, entry_tid=10917' 2024-08-31T22:21:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:26.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=85, tag_tid=11, entry_tid=10917' 2024-08-31T22:21:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10917 2024-08-31T22:21:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 10917 2024-08-31T22:21:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=11052], [object_number=87, tag_tid=11, entry_tid=11051], [object_number=86, tag_tid=11, entry_tid=11050], [object_number=85, tag_tid=11, entry_tid=11049]]], state=connected] 2024-08-31T22:21:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=11052], [object_number=87, tag_tid=11, entry_tid=11051], [object_number=86, tag_tid=11, entry_tid=11050], [object_number=85, tag_tid=11, entry_tid=11049]]], state=connected] 2024-08-31T22:21:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=84, tag_tid=11, entry_tid=11052' 2024-08-31T22:21:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=84, tag_tid=11, entry_tid=11052' 2024-08-31T22:21:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=11052' '!=' 'object_number=85, tag_tid=11, entry_tid=10917' 2024-08-31T22:21:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=11052' = 'object_number=85, tag_tid=11, entry_tid=10917' 2024-08-31T22:21:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=84, tag_tid=11, entry_tid=11052' 2024-08-31T22:21:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=11052' 2024-08-31T22:21:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=11052' 2024-08-31T22:21:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11052 2024-08-31T22:21:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 11052 2024-08-31T22:21:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:27.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:27.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:27.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:27.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:27.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:27.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:27.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:27.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:27.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=11142], [object_number=85, tag_tid=11, entry_tid=11141], [object_number=84, tag_tid=11, entry_tid=11140], [object_number=87, tag_tid=11, entry_tid=11139]]], state=connected] 2024-08-31T22:21:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=11142], [object_number=85, tag_tid=11, entry_tid=11141], [object_number=84, tag_tid=11, entry_tid=11140], [object_number=87, tag_tid=11, entry_tid=11139]]], state=connected] 2024-08-31T22:21:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=86, tag_tid=11, entry_tid=11142' 2024-08-31T22:21:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=86, tag_tid=11, entry_tid=11142' 2024-08-31T22:21:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=11142' '!=' 'object_number=84, tag_tid=11, entry_tid=11052' 2024-08-31T22:21:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=11142' = 'object_number=84, tag_tid=11, entry_tid=11052' 2024-08-31T22:21:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=86, tag_tid=11, entry_tid=11142' 2024-08-31T22:21:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=11142' 2024-08-31T22:21:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:27.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:27.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:27.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=11142' 2024-08-31T22:21:27.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:27.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11142 2024-08-31T22:21:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 11142 2024-08-31T22:21:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:28.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:28.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:28.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:28.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:28.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:28.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:28.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:28.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:28.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:28.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:28.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:28.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:28.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:28.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:28.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=11230], [object_number=85, tag_tid=11, entry_tid=11229], [object_number=84, tag_tid=11, entry_tid=11228], [object_number=87, tag_tid=11, entry_tid=11227]]], state=connected] 2024-08-31T22:21:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:28.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:28.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:28.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:28.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:28.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:28.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:28.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=11230], [object_number=85, tag_tid=11, entry_tid=11229], [object_number=84, tag_tid=11, entry_tid=11228], [object_number=87, tag_tid=11, entry_tid=11227]]], state=connected] 2024-08-31T22:21:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=86, tag_tid=11, entry_tid=11230' 2024-08-31T22:21:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=86, tag_tid=11, entry_tid=11230' 2024-08-31T22:21:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=11230' '!=' 'object_number=86, tag_tid=11, entry_tid=11142' 2024-08-31T22:21:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=11230' = 'object_number=86, tag_tid=11, entry_tid=11142' 2024-08-31T22:21:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=86, tag_tid=11, entry_tid=11230' 2024-08-31T22:21:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=11230' 2024-08-31T22:21:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=11230' 2024-08-31T22:21:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11230 2024-08-31T22:21:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 11230 2024-08-31T22:21:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=87, tag_tid=11, entry_tid=11315], [object_number=86, tag_tid=11, entry_tid=11314], [object_number=85, tag_tid=11, entry_tid=11313], [object_number=84, tag_tid=11, entry_tid=11312]]], state=connected] 2024-08-31T22:21:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=87, tag_tid=11, entry_tid=11315], [object_number=86, tag_tid=11, entry_tid=11314], [object_number=85, tag_tid=11, entry_tid=11313], [object_number=84, tag_tid=11, entry_tid=11312]]], state=connected] 2024-08-31T22:21:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=87, tag_tid=11, entry_tid=11315' 2024-08-31T22:21:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=87, tag_tid=11, entry_tid=11315' 2024-08-31T22:21:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=87, tag_tid=11, entry_tid=11315' '!=' 'object_number=86, tag_tid=11, entry_tid=11230' 2024-08-31T22:21:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=87, tag_tid=11, entry_tid=11315' = 'object_number=86, tag_tid=11, entry_tid=11230' 2024-08-31T22:21:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=87, tag_tid=11, entry_tid=11315' 2024-08-31T22:21:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=87, tag_tid=11, entry_tid=11315' 2024-08-31T22:21:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=87, tag_tid=11, entry_tid=11315' 2024-08-31T22:21:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11315 2024-08-31T22:21:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 11315 2024-08-31T22:21:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=11400], [object_number=87, tag_tid=11, entry_tid=11399], [object_number=86, tag_tid=11, entry_tid=11398], [object_number=85, tag_tid=11, entry_tid=11397]]], state=connected] 2024-08-31T22:21:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=11400], [object_number=87, tag_tid=11, entry_tid=11399], [object_number=86, tag_tid=11, entry_tid=11398], [object_number=85, tag_tid=11, entry_tid=11397]]], state=connected] 2024-08-31T22:21:29.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=84, tag_tid=11, entry_tid=11400' 2024-08-31T22:21:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=84, tag_tid=11, entry_tid=11400' 2024-08-31T22:21:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=11400' '!=' 'object_number=87, tag_tid=11, entry_tid=11315' 2024-08-31T22:21:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=11400' = 'object_number=87, tag_tid=11, entry_tid=11315' 2024-08-31T22:21:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=84, tag_tid=11, entry_tid=11400' 2024-08-31T22:21:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=11400' 2024-08-31T22:21:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=11400' 2024-08-31T22:21:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:29.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11400 2024-08-31T22:21:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 11400 2024-08-31T22:21:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=87, tag_tid=11, entry_tid=11523], [object_number=86, tag_tid=11, entry_tid=11522], [object_number=85, tag_tid=11, entry_tid=11521], [object_number=84, tag_tid=11, entry_tid=11520]]], state=connected] 2024-08-31T22:21:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=87, tag_tid=11, entry_tid=11523], [object_number=86, tag_tid=11, entry_tid=11522], [object_number=85, tag_tid=11, entry_tid=11521], [object_number=84, tag_tid=11, entry_tid=11520]]], state=connected] 2024-08-31T22:21:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=87, tag_tid=11, entry_tid=11523' 2024-08-31T22:21:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=87, tag_tid=11, entry_tid=11523' 2024-08-31T22:21:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=87, tag_tid=11, entry_tid=11523' '!=' 'object_number=84, tag_tid=11, entry_tid=11400' 2024-08-31T22:21:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=87, tag_tid=11, entry_tid=11523' = 'object_number=84, tag_tid=11, entry_tid=11400' 2024-08-31T22:21:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=87, tag_tid=11, entry_tid=11523' 2024-08-31T22:21:30.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:30.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:30.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:30.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=87, tag_tid=11, entry_tid=11523' 2024-08-31T22:21:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:30.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:30.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:30.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:30.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:30.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=87, tag_tid=11, entry_tid=11523' 2024-08-31T22:21:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:30.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11523 2024-08-31T22:21:30.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 11523 2024-08-31T22:21:30.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:30.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=11608], [object_number=87, tag_tid=11, entry_tid=11607], [object_number=86, tag_tid=11, entry_tid=11606], [object_number=85, tag_tid=11, entry_tid=11605]]], state=connected] 2024-08-31T22:21:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=11608], [object_number=87, tag_tid=11, entry_tid=11607], [object_number=86, tag_tid=11, entry_tid=11606], [object_number=85, tag_tid=11, entry_tid=11605]]], state=connected] 2024-08-31T22:21:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=84, tag_tid=11, entry_tid=11608' 2024-08-31T22:21:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=84, tag_tid=11, entry_tid=11608' 2024-08-31T22:21:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=11608' '!=' 'object_number=87, tag_tid=11, entry_tid=11523' 2024-08-31T22:21:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=11608' = 'object_number=87, tag_tid=11, entry_tid=11523' 2024-08-31T22:21:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=84, tag_tid=11, entry_tid=11608' 2024-08-31T22:21:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=11608' 2024-08-31T22:21:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=11608' 2024-08-31T22:21:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11608 2024-08-31T22:21:30.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 11608 2024-08-31T22:21:30.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:30.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:30.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=11694], [object_number=85, tag_tid=11, entry_tid=11693], [object_number=84, tag_tid=11, entry_tid=11692], [object_number=87, tag_tid=11, entry_tid=11691]]], state=connected] 2024-08-31T22:21:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=11694], [object_number=85, tag_tid=11, entry_tid=11693], [object_number=84, tag_tid=11, entry_tid=11692], [object_number=87, tag_tid=11, entry_tid=11691]]], state=connected] 2024-08-31T22:21:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=86, tag_tid=11, entry_tid=11694' 2024-08-31T22:21:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=86, tag_tid=11, entry_tid=11694' 2024-08-31T22:21:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=11694' '!=' 'object_number=84, tag_tid=11, entry_tid=11608' 2024-08-31T22:21:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=11694' = 'object_number=84, tag_tid=11, entry_tid=11608' 2024-08-31T22:21:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=86, tag_tid=11, entry_tid=11694' 2024-08-31T22:21:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=11694' 2024-08-31T22:21:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=11694' 2024-08-31T22:21:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11694 2024-08-31T22:21:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 11694 2024-08-31T22:21:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=11780], [object_number=87, tag_tid=11, entry_tid=11779], [object_number=86, tag_tid=11, entry_tid=11778], [object_number=85, tag_tid=11, entry_tid=11777]]], state=connected] 2024-08-31T22:21:31.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=11780], [object_number=87, tag_tid=11, entry_tid=11779], [object_number=86, tag_tid=11, entry_tid=11778], [object_number=85, tag_tid=11, entry_tid=11777]]], state=connected] 2024-08-31T22:21:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=84, tag_tid=11, entry_tid=11780' 2024-08-31T22:21:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=84, tag_tid=11, entry_tid=11780' 2024-08-31T22:21:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=11780' '!=' 'object_number=86, tag_tid=11, entry_tid=11694' 2024-08-31T22:21:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=11780' = 'object_number=86, tag_tid=11, entry_tid=11694' 2024-08-31T22:21:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=84, tag_tid=11, entry_tid=11780' 2024-08-31T22:21:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=11780' 2024-08-31T22:21:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:31.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:31.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:31.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:31.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:31.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:31.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=11780' 2024-08-31T22:21:31.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:31.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11780 2024-08-31T22:21:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 11780 2024-08-31T22:21:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:31.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:31.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:31.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:31.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=87, tag_tid=11, entry_tid=11875], [object_number=86, tag_tid=11, entry_tid=11874], [object_number=85, tag_tid=11, entry_tid=11873], [object_number=84, tag_tid=11, entry_tid=11872]]], state=connected] 2024-08-31T22:21:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=87, tag_tid=11, entry_tid=11875], [object_number=86, tag_tid=11, entry_tid=11874], [object_number=85, tag_tid=11, entry_tid=11873], [object_number=84, tag_tid=11, entry_tid=11872]]], state=connected] 2024-08-31T22:21:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=87, tag_tid=11, entry_tid=11875' 2024-08-31T22:21:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=87, tag_tid=11, entry_tid=11875' 2024-08-31T22:21:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=87, tag_tid=11, entry_tid=11875' '!=' 'object_number=84, tag_tid=11, entry_tid=11780' 2024-08-31T22:21:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=87, tag_tid=11, entry_tid=11875' = 'object_number=84, tag_tid=11, entry_tid=11780' 2024-08-31T22:21:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=87, tag_tid=11, entry_tid=11875' 2024-08-31T22:21:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:32.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:32.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:32.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=87, tag_tid=11, entry_tid=11875' 2024-08-31T22:21:32.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:32.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=87, tag_tid=11, entry_tid=11875' 2024-08-31T22:21:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:32.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11875 2024-08-31T22:21:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 11875 2024-08-31T22:21:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:32.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:32.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=11968], [object_number=87, tag_tid=11, entry_tid=11967], [object_number=86, tag_tid=11, entry_tid=11966], [object_number=85, tag_tid=11, entry_tid=11965]]], state=connected] 2024-08-31T22:21:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=11968], [object_number=87, tag_tid=11, entry_tid=11967], [object_number=86, tag_tid=11, entry_tid=11966], [object_number=85, tag_tid=11, entry_tid=11965]]], state=connected] 2024-08-31T22:21:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=84, tag_tid=11, entry_tid=11968' 2024-08-31T22:21:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=84, tag_tid=11, entry_tid=11968' 2024-08-31T22:21:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=11968' '!=' 'object_number=87, tag_tid=11, entry_tid=11875' 2024-08-31T22:21:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=11968' = 'object_number=87, tag_tid=11, entry_tid=11875' 2024-08-31T22:21:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=84, tag_tid=11, entry_tid=11968' 2024-08-31T22:21:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=11968' 2024-08-31T22:21:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=11968' 2024-08-31T22:21:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11968 2024-08-31T22:21:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 11968 2024-08-31T22:21:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:32.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:32.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:32.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:32.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:32.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:32.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:32.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:32.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:32.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:32.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:32.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:32.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:32.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:32.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:32.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=85, tag_tid=11, entry_tid=12077], [object_number=84, tag_tid=11, entry_tid=12076], [object_number=87, tag_tid=11, entry_tid=12075], [object_number=86, tag_tid=11, entry_tid=12074]]], state=connected] 2024-08-31T22:21:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=85, tag_tid=11, entry_tid=12077], [object_number=84, tag_tid=11, entry_tid=12076], [object_number=87, tag_tid=11, entry_tid=12075], [object_number=86, tag_tid=11, entry_tid=12074]]], state=connected] 2024-08-31T22:21:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=85, tag_tid=11, entry_tid=12077' 2024-08-31T22:21:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=85, tag_tid=11, entry_tid=12077' 2024-08-31T22:21:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=85, tag_tid=11, entry_tid=12077' '!=' 'object_number=84, tag_tid=11, entry_tid=11968' 2024-08-31T22:21:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=85, tag_tid=11, entry_tid=12077' = 'object_number=84, tag_tid=11, entry_tid=11968' 2024-08-31T22:21:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=85, tag_tid=11, entry_tid=12077' 2024-08-31T22:21:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=85, tag_tid=11, entry_tid=12077' 2024-08-31T22:21:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=85, tag_tid=11, entry_tid=12077' 2024-08-31T22:21:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12077 2024-08-31T22:21:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 12077 2024-08-31T22:21:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:33.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:33.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:33.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:33.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:33.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:33.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=87, tag_tid=11, entry_tid=12179], [object_number=86, tag_tid=11, entry_tid=12178], [object_number=85, tag_tid=11, entry_tid=12177], [object_number=84, tag_tid=11, entry_tid=12176]]], state=connected] 2024-08-31T22:21:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=87, tag_tid=11, entry_tid=12179], [object_number=86, tag_tid=11, entry_tid=12178], [object_number=85, tag_tid=11, entry_tid=12177], [object_number=84, tag_tid=11, entry_tid=12176]]], state=connected] 2024-08-31T22:21:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=87, tag_tid=11, entry_tid=12179' 2024-08-31T22:21:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=87, tag_tid=11, entry_tid=12179' 2024-08-31T22:21:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=87, tag_tid=11, entry_tid=12179' '!=' 'object_number=85, tag_tid=11, entry_tid=12077' 2024-08-31T22:21:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=87, tag_tid=11, entry_tid=12179' = 'object_number=85, tag_tid=11, entry_tid=12077' 2024-08-31T22:21:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=87, tag_tid=11, entry_tid=12179' 2024-08-31T22:21:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=87, tag_tid=11, entry_tid=12179' 2024-08-31T22:21:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=87, tag_tid=11, entry_tid=12179' 2024-08-31T22:21:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12179 2024-08-31T22:21:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 12179 2024-08-31T22:21:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=12274], [object_number=85, tag_tid=11, entry_tid=12273], [object_number=84, tag_tid=11, entry_tid=12272], [object_number=87, tag_tid=11, entry_tid=12271]]], state=connected] 2024-08-31T22:21:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=86, tag_tid=11, entry_tid=12274], [object_number=85, tag_tid=11, entry_tid=12273], [object_number=84, tag_tid=11, entry_tid=12272], [object_number=87, tag_tid=11, entry_tid=12271]]], state=connected] 2024-08-31T22:21:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=86, tag_tid=11, entry_tid=12274' 2024-08-31T22:21:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=86, tag_tid=11, entry_tid=12274' 2024-08-31T22:21:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=12274' '!=' 'object_number=87, tag_tid=11, entry_tid=12179' 2024-08-31T22:21:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=86, tag_tid=11, entry_tid=12274' = 'object_number=87, tag_tid=11, entry_tid=12179' 2024-08-31T22:21:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=86, tag_tid=11, entry_tid=12274' 2024-08-31T22:21:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=12274' 2024-08-31T22:21:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:34.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:34.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=86, tag_tid=11, entry_tid=12274' 2024-08-31T22:21:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12274 2024-08-31T22:21:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 12274 2024-08-31T22:21:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=12368], [object_number=87, tag_tid=11, entry_tid=12367], [object_number=86, tag_tid=11, entry_tid=12366], [object_number=85, tag_tid=11, entry_tid=12365]]], state=connected] 2024-08-31T22:21:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=12368], [object_number=87, tag_tid=11, entry_tid=12367], [object_number=86, tag_tid=11, entry_tid=12366], [object_number=85, tag_tid=11, entry_tid=12365]]], state=connected] 2024-08-31T22:21:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=84, tag_tid=11, entry_tid=12368' 2024-08-31T22:21:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=84, tag_tid=11, entry_tid=12368' 2024-08-31T22:21:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=12368' '!=' 'object_number=86, tag_tid=11, entry_tid=12274' 2024-08-31T22:21:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=12368' = 'object_number=86, tag_tid=11, entry_tid=12274' 2024-08-31T22:21:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=84, tag_tid=11, entry_tid=12368' 2024-08-31T22:21:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=12368' 2024-08-31T22:21:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=12368' 2024-08-31T22:21:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12368 2024-08-31T22:21:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 12368 2024-08-31T22:21:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:34.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=85, tag_tid=11, entry_tid=12461], [object_number=84, tag_tid=11, entry_tid=12460], [object_number=87, tag_tid=11, entry_tid=12459], [object_number=86, tag_tid=11, entry_tid=12458]]], state=connected] 2024-08-31T22:21:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=85, tag_tid=11, entry_tid=12461], [object_number=84, tag_tid=11, entry_tid=12460], [object_number=87, tag_tid=11, entry_tid=12459], [object_number=86, tag_tid=11, entry_tid=12458]]], state=connected] 2024-08-31T22:21:35.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=85, tag_tid=11, entry_tid=12461' 2024-08-31T22:21:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=85, tag_tid=11, entry_tid=12461' 2024-08-31T22:21:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=85, tag_tid=11, entry_tid=12461' '!=' 'object_number=84, tag_tid=11, entry_tid=12368' 2024-08-31T22:21:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=85, tag_tid=11, entry_tid=12461' = 'object_number=84, tag_tid=11, entry_tid=12368' 2024-08-31T22:21:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=85, tag_tid=11, entry_tid=12461' 2024-08-31T22:21:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=85, tag_tid=11, entry_tid=12461' 2024-08-31T22:21:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=85, tag_tid=11, entry_tid=12461' 2024-08-31T22:21:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12461 2024-08-31T22:21:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 12461 2024-08-31T22:21:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=12612], [object_number=87, tag_tid=11, entry_tid=12611], [object_number=86, tag_tid=11, entry_tid=12610], [object_number=85, tag_tid=11, entry_tid=12609]]], state=connected] 2024-08-31T22:21:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=84, tag_tid=11, entry_tid=12612], [object_number=87, tag_tid=11, entry_tid=12611], [object_number=86, tag_tid=11, entry_tid=12610], [object_number=85, tag_tid=11, entry_tid=12609]]], state=connected] 2024-08-31T22:21:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=84, tag_tid=11, entry_tid=12612' 2024-08-31T22:21:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=84, tag_tid=11, entry_tid=12612' 2024-08-31T22:21:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=12612' '!=' 'object_number=85, tag_tid=11, entry_tid=12461' 2024-08-31T22:21:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=84, tag_tid=11, entry_tid=12612' = 'object_number=85, tag_tid=11, entry_tid=12461' 2024-08-31T22:21:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=84, tag_tid=11, entry_tid=12612' 2024-08-31T22:21:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=12612' 2024-08-31T22:21:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=84, tag_tid=11, entry_tid=12612' 2024-08-31T22:21:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12612 2024-08-31T22:21:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 12612 2024-08-31T22:21:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:35.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:35.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=12635], [object_number=90, tag_tid=11, entry_tid=12634], [object_number=89, tag_tid=11, entry_tid=12633], [object_number=84, tag_tid=11, entry_tid=12632]]], state=connected] 2024-08-31T22:21:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 21 2024-08-31T22:21:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=12635], [object_number=90, tag_tid=11, entry_tid=12634], [object_number=89, tag_tid=11, entry_tid=12633], [object_number=84, tag_tid=11, entry_tid=12632]]], state=connected] 2024-08-31T22:21:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=91, tag_tid=11, entry_tid=12635' 2024-08-31T22:21:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=91, tag_tid=11, entry_tid=12635' 2024-08-31T22:21:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=12635' '!=' 'object_number=84, tag_tid=11, entry_tid=12612' 2024-08-31T22:21:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=12635' = 'object_number=84, tag_tid=11, entry_tid=12612' 2024-08-31T22:21:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=91, tag_tid=11, entry_tid=12635' 2024-08-31T22:21:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=12635' 2024-08-31T22:21:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=12635' 2024-08-31T22:21:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:36.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12635 2024-08-31T22:21:36.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 12635 2024-08-31T22:21:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=90, tag_tid=11, entry_tid=12730], [object_number=89, tag_tid=11, entry_tid=12729], [object_number=88, tag_tid=11, entry_tid=12728], [object_number=91, tag_tid=11, entry_tid=12727]]], state=connected] 2024-08-31T22:21:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:36.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:36.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:36.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:36.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=90, tag_tid=11, entry_tid=12730], [object_number=89, tag_tid=11, entry_tid=12729], [object_number=88, tag_tid=11, entry_tid=12728], [object_number=91, tag_tid=11, entry_tid=12727]]], state=connected] 2024-08-31T22:21:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=90, tag_tid=11, entry_tid=12730' 2024-08-31T22:21:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=90, tag_tid=11, entry_tid=12730' 2024-08-31T22:21:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=90, tag_tid=11, entry_tid=12730' '!=' 'object_number=91, tag_tid=11, entry_tid=12635' 2024-08-31T22:21:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=90, tag_tid=11, entry_tid=12730' = 'object_number=91, tag_tid=11, entry_tid=12635' 2024-08-31T22:21:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=90, tag_tid=11, entry_tid=12730' 2024-08-31T22:21:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=90, tag_tid=11, entry_tid=12730' 2024-08-31T22:21:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=90, tag_tid=11, entry_tid=12730' 2024-08-31T22:21:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12730 2024-08-31T22:21:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 12730 2024-08-31T22:21:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=88, tag_tid=11, entry_tid=12832], [object_number=91, tag_tid=11, entry_tid=12831], [object_number=90, tag_tid=11, entry_tid=12830], [object_number=89, tag_tid=11, entry_tid=12829]]], state=connected] 2024-08-31T22:21:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=88, tag_tid=11, entry_tid=12832], [object_number=91, tag_tid=11, entry_tid=12831], [object_number=90, tag_tid=11, entry_tid=12830], [object_number=89, tag_tid=11, entry_tid=12829]]], state=connected] 2024-08-31T22:21:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=88, tag_tid=11, entry_tid=12832' 2024-08-31T22:21:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=88, tag_tid=11, entry_tid=12832' 2024-08-31T22:21:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=88, tag_tid=11, entry_tid=12832' '!=' 'object_number=90, tag_tid=11, entry_tid=12730' 2024-08-31T22:21:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=88, tag_tid=11, entry_tid=12832' = 'object_number=90, tag_tid=11, entry_tid=12730' 2024-08-31T22:21:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=88, tag_tid=11, entry_tid=12832' 2024-08-31T22:21:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=88, tag_tid=11, entry_tid=12832' 2024-08-31T22:21:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=88, tag_tid=11, entry_tid=12832' 2024-08-31T22:21:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:37.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12832 2024-08-31T22:21:37.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 12832 2024-08-31T22:21:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:37.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:37.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:37.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:37.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:37.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:37.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:37.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=12923], [object_number=90, tag_tid=11, entry_tid=12922], [object_number=89, tag_tid=11, entry_tid=12921], [object_number=88, tag_tid=11, entry_tid=12920]]], state=connected] 2024-08-31T22:21:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=12923], [object_number=90, tag_tid=11, entry_tid=12922], [object_number=89, tag_tid=11, entry_tid=12921], [object_number=88, tag_tid=11, entry_tid=12920]]], state=connected] 2024-08-31T22:21:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=91, tag_tid=11, entry_tid=12923' 2024-08-31T22:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=91, tag_tid=11, entry_tid=12923' 2024-08-31T22:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=12923' '!=' 'object_number=88, tag_tid=11, entry_tid=12832' 2024-08-31T22:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=12923' = 'object_number=88, tag_tid=11, entry_tid=12832' 2024-08-31T22:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=91, tag_tid=11, entry_tid=12923' 2024-08-31T22:21:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=12923' 2024-08-31T22:21:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=12923' 2024-08-31T22:21:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12923 2024-08-31T22:21:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 12923 2024-08-31T22:21:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:37.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:37.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:37.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:37.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=88, tag_tid=11, entry_tid=13016], [object_number=91, tag_tid=11, entry_tid=13015], [object_number=90, tag_tid=11, entry_tid=13014], [object_number=89, tag_tid=11, entry_tid=13013]]], state=connected] 2024-08-31T22:21:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=88, tag_tid=11, entry_tid=13016], [object_number=91, tag_tid=11, entry_tid=13015], [object_number=90, tag_tid=11, entry_tid=13014], [object_number=89, tag_tid=11, entry_tid=13013]]], state=connected] 2024-08-31T22:21:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=88, tag_tid=11, entry_tid=13016' 2024-08-31T22:21:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=88, tag_tid=11, entry_tid=13016' 2024-08-31T22:21:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=88, tag_tid=11, entry_tid=13016' '!=' 'object_number=91, tag_tid=11, entry_tid=12923' 2024-08-31T22:21:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=88, tag_tid=11, entry_tid=13016' = 'object_number=91, tag_tid=11, entry_tid=12923' 2024-08-31T22:21:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=88, tag_tid=11, entry_tid=13016' 2024-08-31T22:21:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=88, tag_tid=11, entry_tid=13016' 2024-08-31T22:21:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=88, tag_tid=11, entry_tid=13016' 2024-08-31T22:21:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13016 2024-08-31T22:21:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 13016 2024-08-31T22:21:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=90, tag_tid=11, entry_tid=13122], [object_number=89, tag_tid=11, entry_tid=13121], [object_number=88, tag_tid=11, entry_tid=13120], [object_number=91, tag_tid=11, entry_tid=13119]]], state=connected] 2024-08-31T22:21:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:38.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=90, tag_tid=11, entry_tid=13122], [object_number=89, tag_tid=11, entry_tid=13121], [object_number=88, tag_tid=11, entry_tid=13120], [object_number=91, tag_tid=11, entry_tid=13119]]], state=connected] 2024-08-31T22:21:38.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=90, tag_tid=11, entry_tid=13122' 2024-08-31T22:21:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=90, tag_tid=11, entry_tid=13122' 2024-08-31T22:21:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=90, tag_tid=11, entry_tid=13122' '!=' 'object_number=88, tag_tid=11, entry_tid=13016' 2024-08-31T22:21:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=90, tag_tid=11, entry_tid=13122' = 'object_number=88, tag_tid=11, entry_tid=13016' 2024-08-31T22:21:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=90, tag_tid=11, entry_tid=13122' 2024-08-31T22:21:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=90, tag_tid=11, entry_tid=13122' 2024-08-31T22:21:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=90, tag_tid=11, entry_tid=13122' 2024-08-31T22:21:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13122 2024-08-31T22:21:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 13122 2024-08-31T22:21:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:39.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:39.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:39.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:39.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:39.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:39.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=90, tag_tid=11, entry_tid=13238], [object_number=89, tag_tid=11, entry_tid=13237], [object_number=88, tag_tid=11, entry_tid=13236], [object_number=91, tag_tid=11, entry_tid=13235]]], state=connected] 2024-08-31T22:21:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:39.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:39.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:39.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=90, tag_tid=11, entry_tid=13238], [object_number=89, tag_tid=11, entry_tid=13237], [object_number=88, tag_tid=11, entry_tid=13236], [object_number=91, tag_tid=11, entry_tid=13235]]], state=connected] 2024-08-31T22:21:39.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=90, tag_tid=11, entry_tid=13238' 2024-08-31T22:21:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=90, tag_tid=11, entry_tid=13238' 2024-08-31T22:21:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=90, tag_tid=11, entry_tid=13238' '!=' 'object_number=90, tag_tid=11, entry_tid=13122' 2024-08-31T22:21:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=90, tag_tid=11, entry_tid=13238' = 'object_number=90, tag_tid=11, entry_tid=13122' 2024-08-31T22:21:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=90, tag_tid=11, entry_tid=13238' 2024-08-31T22:21:39.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:39.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:39.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:39.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=90, tag_tid=11, entry_tid=13238' 2024-08-31T22:21:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=90, tag_tid=11, entry_tid=13238' 2024-08-31T22:21:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13238 2024-08-31T22:21:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 13238 2024-08-31T22:21:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=90, tag_tid=11, entry_tid=13350], [object_number=89, tag_tid=11, entry_tid=13349], [object_number=88, tag_tid=11, entry_tid=13348], [object_number=91, tag_tid=11, entry_tid=13347]]], state=connected] 2024-08-31T22:21:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=90, tag_tid=11, entry_tid=13350], [object_number=89, tag_tid=11, entry_tid=13349], [object_number=88, tag_tid=11, entry_tid=13348], [object_number=91, tag_tid=11, entry_tid=13347]]], state=connected] 2024-08-31T22:21:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=90, tag_tid=11, entry_tid=13350' 2024-08-31T22:21:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=90, tag_tid=11, entry_tid=13350' 2024-08-31T22:21:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=90, tag_tid=11, entry_tid=13350' '!=' 'object_number=90, tag_tid=11, entry_tid=13238' 2024-08-31T22:21:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=90, tag_tid=11, entry_tid=13350' = 'object_number=90, tag_tid=11, entry_tid=13238' 2024-08-31T22:21:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=90, tag_tid=11, entry_tid=13350' 2024-08-31T22:21:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=90, tag_tid=11, entry_tid=13350' 2024-08-31T22:21:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=90, tag_tid=11, entry_tid=13350' 2024-08-31T22:21:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13350 2024-08-31T22:21:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 13350 2024-08-31T22:21:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=13451], [object_number=90, tag_tid=11, entry_tid=13450], [object_number=89, tag_tid=11, entry_tid=13449], [object_number=88, tag_tid=11, entry_tid=13448]]], state=connected] 2024-08-31T22:21:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=13451], [object_number=90, tag_tid=11, entry_tid=13450], [object_number=89, tag_tid=11, entry_tid=13449], [object_number=88, tag_tid=11, entry_tid=13448]]], state=connected] 2024-08-31T22:21:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=91, tag_tid=11, entry_tid=13451' 2024-08-31T22:21:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=91, tag_tid=11, entry_tid=13451' 2024-08-31T22:21:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=13451' '!=' 'object_number=90, tag_tid=11, entry_tid=13350' 2024-08-31T22:21:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=13451' = 'object_number=90, tag_tid=11, entry_tid=13350' 2024-08-31T22:21:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=91, tag_tid=11, entry_tid=13451' 2024-08-31T22:21:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=13451' 2024-08-31T22:21:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=13451' 2024-08-31T22:21:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13451 2024-08-31T22:21:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 13451 2024-08-31T22:21:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=89, tag_tid=11, entry_tid=13553], [object_number=88, tag_tid=11, entry_tid=13552], [object_number=91, tag_tid=11, entry_tid=13551], [object_number=90, tag_tid=11, entry_tid=13550]]], state=connected] 2024-08-31T22:21:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:40.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:40.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:40.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=89, tag_tid=11, entry_tid=13553], [object_number=88, tag_tid=11, entry_tid=13552], [object_number=91, tag_tid=11, entry_tid=13551], [object_number=90, tag_tid=11, entry_tid=13550]]], state=connected] 2024-08-31T22:21:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:40.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=89, tag_tid=11, entry_tid=13553' 2024-08-31T22:21:40.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=89, tag_tid=11, entry_tid=13553' 2024-08-31T22:21:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=89, tag_tid=11, entry_tid=13553' '!=' 'object_number=91, tag_tid=11, entry_tid=13451' 2024-08-31T22:21:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=89, tag_tid=11, entry_tid=13553' = 'object_number=91, tag_tid=11, entry_tid=13451' 2024-08-31T22:21:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=89, tag_tid=11, entry_tid=13553' 2024-08-31T22:21:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:40.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:40.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=89, tag_tid=11, entry_tid=13553' 2024-08-31T22:21:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=89, tag_tid=11, entry_tid=13553' 2024-08-31T22:21:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13553 2024-08-31T22:21:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 13553 2024-08-31T22:21:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=90, tag_tid=11, entry_tid=13662], [object_number=89, tag_tid=11, entry_tid=13661], [object_number=88, tag_tid=11, entry_tid=13660], [object_number=91, tag_tid=11, entry_tid=13659]]], state=connected] 2024-08-31T22:21:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:41.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=90, tag_tid=11, entry_tid=13662], [object_number=89, tag_tid=11, entry_tid=13661], [object_number=88, tag_tid=11, entry_tid=13660], [object_number=91, tag_tid=11, entry_tid=13659]]], state=connected] 2024-08-31T22:21:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=90, tag_tid=11, entry_tid=13662' 2024-08-31T22:21:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=90, tag_tid=11, entry_tid=13662' 2024-08-31T22:21:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=90, tag_tid=11, entry_tid=13662' '!=' 'object_number=89, tag_tid=11, entry_tid=13553' 2024-08-31T22:21:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=90, tag_tid=11, entry_tid=13662' = 'object_number=89, tag_tid=11, entry_tid=13553' 2024-08-31T22:21:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=90, tag_tid=11, entry_tid=13662' 2024-08-31T22:21:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=90, tag_tid=11, entry_tid=13662' 2024-08-31T22:21:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=90, tag_tid=11, entry_tid=13662' 2024-08-31T22:21:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13662 2024-08-31T22:21:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 13662 2024-08-31T22:21:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:41.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:41.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:41.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:41.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:41.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:41.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:41.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:41.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:41.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:41.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:41.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:41.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:41.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:41.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:41.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=89, tag_tid=11, entry_tid=13769], [object_number=88, tag_tid=11, entry_tid=13768], [object_number=91, tag_tid=11, entry_tid=13767], [object_number=90, tag_tid=11, entry_tid=13766]]], state=connected] 2024-08-31T22:21:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=89, tag_tid=11, entry_tid=13769], [object_number=88, tag_tid=11, entry_tid=13768], [object_number=91, tag_tid=11, entry_tid=13767], [object_number=90, tag_tid=11, entry_tid=13766]]], state=connected] 2024-08-31T22:21:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=89, tag_tid=11, entry_tid=13769' 2024-08-31T22:21:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=89, tag_tid=11, entry_tid=13769' 2024-08-31T22:21:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=89, tag_tid=11, entry_tid=13769' '!=' 'object_number=90, tag_tid=11, entry_tid=13662' 2024-08-31T22:21:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=89, tag_tid=11, entry_tid=13769' = 'object_number=90, tag_tid=11, entry_tid=13662' 2024-08-31T22:21:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=89, tag_tid=11, entry_tid=13769' 2024-08-31T22:21:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=89, tag_tid=11, entry_tid=13769' 2024-08-31T22:21:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=89, tag_tid=11, entry_tid=13769' 2024-08-31T22:21:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13769 2024-08-31T22:21:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 13769 2024-08-31T22:21:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=90, tag_tid=11, entry_tid=13862], [object_number=89, tag_tid=11, entry_tid=13861], [object_number=88, tag_tid=11, entry_tid=13860], [object_number=91, tag_tid=11, entry_tid=13859]]], state=connected] 2024-08-31T22:21:42.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=90, tag_tid=11, entry_tid=13862], [object_number=89, tag_tid=11, entry_tid=13861], [object_number=88, tag_tid=11, entry_tid=13860], [object_number=91, tag_tid=11, entry_tid=13859]]], state=connected] 2024-08-31T22:21:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=90, tag_tid=11, entry_tid=13862' 2024-08-31T22:21:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=90, tag_tid=11, entry_tid=13862' 2024-08-31T22:21:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=90, tag_tid=11, entry_tid=13862' '!=' 'object_number=89, tag_tid=11, entry_tid=13769' 2024-08-31T22:21:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=90, tag_tid=11, entry_tid=13862' = 'object_number=89, tag_tid=11, entry_tid=13769' 2024-08-31T22:21:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=90, tag_tid=11, entry_tid=13862' 2024-08-31T22:21:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=90, tag_tid=11, entry_tid=13862' 2024-08-31T22:21:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=90, tag_tid=11, entry_tid=13862' 2024-08-31T22:21:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13862 2024-08-31T22:21:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 13862 2024-08-31T22:21:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=88, tag_tid=11, entry_tid=13952], [object_number=91, tag_tid=11, entry_tid=13951], [object_number=90, tag_tid=11, entry_tid=13950], [object_number=89, tag_tid=11, entry_tid=13949]]], state=connected] 2024-08-31T22:21:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=88, tag_tid=11, entry_tid=13952], [object_number=91, tag_tid=11, entry_tid=13951], [object_number=90, tag_tid=11, entry_tid=13950], [object_number=89, tag_tid=11, entry_tid=13949]]], state=connected] 2024-08-31T22:21:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=88, tag_tid=11, entry_tid=13952' 2024-08-31T22:21:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=88, tag_tid=11, entry_tid=13952' 2024-08-31T22:21:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=88, tag_tid=11, entry_tid=13952' '!=' 'object_number=90, tag_tid=11, entry_tid=13862' 2024-08-31T22:21:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=88, tag_tid=11, entry_tid=13952' = 'object_number=90, tag_tid=11, entry_tid=13862' 2024-08-31T22:21:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=88, tag_tid=11, entry_tid=13952' 2024-08-31T22:21:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=88, tag_tid=11, entry_tid=13952' 2024-08-31T22:21:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=88, tag_tid=11, entry_tid=13952' 2024-08-31T22:21:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13952 2024-08-31T22:21:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 13952 2024-08-31T22:21:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:43.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:43.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:43.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:43.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=88, tag_tid=11, entry_tid=14036], [object_number=91, tag_tid=11, entry_tid=14035], [object_number=90, tag_tid=11, entry_tid=14034], [object_number=89, tag_tid=11, entry_tid=14033]]], state=connected] 2024-08-31T22:21:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=88, tag_tid=11, entry_tid=14036], [object_number=91, tag_tid=11, entry_tid=14035], [object_number=90, tag_tid=11, entry_tid=14034], [object_number=89, tag_tid=11, entry_tid=14033]]], state=connected] 2024-08-31T22:21:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=88, tag_tid=11, entry_tid=14036' 2024-08-31T22:21:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=88, tag_tid=11, entry_tid=14036' 2024-08-31T22:21:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=88, tag_tid=11, entry_tid=14036' '!=' 'object_number=88, tag_tid=11, entry_tid=13952' 2024-08-31T22:21:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=88, tag_tid=11, entry_tid=14036' = 'object_number=88, tag_tid=11, entry_tid=13952' 2024-08-31T22:21:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=88, tag_tid=11, entry_tid=14036' 2024-08-31T22:21:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=88, tag_tid=11, entry_tid=14036' 2024-08-31T22:21:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=88, tag_tid=11, entry_tid=14036' 2024-08-31T22:21:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14036 2024-08-31T22:21:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 14036 2024-08-31T22:21:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=89, tag_tid=11, entry_tid=14169], [object_number=88, tag_tid=11, entry_tid=14168], [object_number=91, tag_tid=11, entry_tid=14167], [object_number=90, tag_tid=11, entry_tid=14166]]], state=connected] 2024-08-31T22:21:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:43.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:43.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:43.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=89, tag_tid=11, entry_tid=14169], [object_number=88, tag_tid=11, entry_tid=14168], [object_number=91, tag_tid=11, entry_tid=14167], [object_number=90, tag_tid=11, entry_tid=14166]]], state=connected] 2024-08-31T22:21:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=89, tag_tid=11, entry_tid=14169' 2024-08-31T22:21:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=89, tag_tid=11, entry_tid=14169' 2024-08-31T22:21:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=89, tag_tid=11, entry_tid=14169' '!=' 'object_number=88, tag_tid=11, entry_tid=14036' 2024-08-31T22:21:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=89, tag_tid=11, entry_tid=14169' = 'object_number=88, tag_tid=11, entry_tid=14036' 2024-08-31T22:21:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=89, tag_tid=11, entry_tid=14169' 2024-08-31T22:21:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:43.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=89, tag_tid=11, entry_tid=14169' 2024-08-31T22:21:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=89, tag_tid=11, entry_tid=14169' 2024-08-31T22:21:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:43.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14169 2024-08-31T22:21:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 14169 2024-08-31T22:21:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=88, tag_tid=11, entry_tid=14252], [object_number=91, tag_tid=11, entry_tid=14251], [object_number=90, tag_tid=11, entry_tid=14250], [object_number=89, tag_tid=11, entry_tid=14249]]], state=connected] 2024-08-31T22:21:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=88, tag_tid=11, entry_tid=14252], [object_number=91, tag_tid=11, entry_tid=14251], [object_number=90, tag_tid=11, entry_tid=14250], [object_number=89, tag_tid=11, entry_tid=14249]]], state=connected] 2024-08-31T22:21:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:44.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=88, tag_tid=11, entry_tid=14252' 2024-08-31T22:21:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=88, tag_tid=11, entry_tid=14252' 2024-08-31T22:21:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=88, tag_tid=11, entry_tid=14252' '!=' 'object_number=89, tag_tid=11, entry_tid=14169' 2024-08-31T22:21:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=88, tag_tid=11, entry_tid=14252' = 'object_number=89, tag_tid=11, entry_tid=14169' 2024-08-31T22:21:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=88, tag_tid=11, entry_tid=14252' 2024-08-31T22:21:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=88, tag_tid=11, entry_tid=14252' 2024-08-31T22:21:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=88, tag_tid=11, entry_tid=14252' 2024-08-31T22:21:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14252 2024-08-31T22:21:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 14252 2024-08-31T22:21:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:44.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:44.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=89, tag_tid=11, entry_tid=14365], [object_number=88, tag_tid=11, entry_tid=14364], [object_number=91, tag_tid=11, entry_tid=14363], [object_number=90, tag_tid=11, entry_tid=14362]]], state=connected] 2024-08-31T22:21:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=89, tag_tid=11, entry_tid=14365], [object_number=88, tag_tid=11, entry_tid=14364], [object_number=91, tag_tid=11, entry_tid=14363], [object_number=90, tag_tid=11, entry_tid=14362]]], state=connected] 2024-08-31T22:21:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=89, tag_tid=11, entry_tid=14365' 2024-08-31T22:21:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=89, tag_tid=11, entry_tid=14365' 2024-08-31T22:21:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=89, tag_tid=11, entry_tid=14365' '!=' 'object_number=88, tag_tid=11, entry_tid=14252' 2024-08-31T22:21:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=89, tag_tid=11, entry_tid=14365' = 'object_number=88, tag_tid=11, entry_tid=14252' 2024-08-31T22:21:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=89, tag_tid=11, entry_tid=14365' 2024-08-31T22:21:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=89, tag_tid=11, entry_tid=14365' 2024-08-31T22:21:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=89, tag_tid=11, entry_tid=14365' 2024-08-31T22:21:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14365 2024-08-31T22:21:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 14365 2024-08-31T22:21:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:45.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:45.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:45.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:45.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:45.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:45.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:45.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:45.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:45.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:45.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:45.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:45.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:45.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:45.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=14475], [object_number=90, tag_tid=11, entry_tid=14474], [object_number=89, tag_tid=11, entry_tid=14473], [object_number=88, tag_tid=11, entry_tid=14472]]], state=connected] 2024-08-31T22:21:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:45.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:45.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:45.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:45.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=14475], [object_number=90, tag_tid=11, entry_tid=14474], [object_number=89, tag_tid=11, entry_tid=14473], [object_number=88, tag_tid=11, entry_tid=14472]]], state=connected] 2024-08-31T22:21:45.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=91, tag_tid=11, entry_tid=14475' 2024-08-31T22:21:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=91, tag_tid=11, entry_tid=14475' 2024-08-31T22:21:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=14475' '!=' 'object_number=89, tag_tid=11, entry_tid=14365' 2024-08-31T22:21:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=14475' = 'object_number=89, tag_tid=11, entry_tid=14365' 2024-08-31T22:21:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=91, tag_tid=11, entry_tid=14475' 2024-08-31T22:21:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:45.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:45.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=14475' 2024-08-31T22:21:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=14475' 2024-08-31T22:21:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14475 2024-08-31T22:21:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 14475 2024-08-31T22:21:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:45.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:45.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:45.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:45.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:45.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:45.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=89, tag_tid=11, entry_tid=14581], [object_number=88, tag_tid=11, entry_tid=14580], [object_number=91, tag_tid=11, entry_tid=14579], [object_number=90, tag_tid=11, entry_tid=14578]]], state=connected] 2024-08-31T22:21:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:45.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=89, tag_tid=11, entry_tid=14581], [object_number=88, tag_tid=11, entry_tid=14580], [object_number=91, tag_tid=11, entry_tid=14579], [object_number=90, tag_tid=11, entry_tid=14578]]], state=connected] 2024-08-31T22:21:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=89, tag_tid=11, entry_tid=14581' 2024-08-31T22:21:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=89, tag_tid=11, entry_tid=14581' 2024-08-31T22:21:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=89, tag_tid=11, entry_tid=14581' '!=' 'object_number=91, tag_tid=11, entry_tid=14475' 2024-08-31T22:21:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=89, tag_tid=11, entry_tid=14581' = 'object_number=91, tag_tid=11, entry_tid=14475' 2024-08-31T22:21:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=89, tag_tid=11, entry_tid=14581' 2024-08-31T22:21:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=89, tag_tid=11, entry_tid=14581' 2024-08-31T22:21:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=89, tag_tid=11, entry_tid=14581' 2024-08-31T22:21:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14581 2024-08-31T22:21:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 14581 2024-08-31T22:21:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=88, tag_tid=11, entry_tid=14668], [object_number=91, tag_tid=11, entry_tid=14667], [object_number=90, tag_tid=11, entry_tid=14666], [object_number=89, tag_tid=11, entry_tid=14665]]], state=connected] 2024-08-31T22:21:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=88, tag_tid=11, entry_tid=14668], [object_number=91, tag_tid=11, entry_tid=14667], [object_number=90, tag_tid=11, entry_tid=14666], [object_number=89, tag_tid=11, entry_tid=14665]]], state=connected] 2024-08-31T22:21:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=88, tag_tid=11, entry_tid=14668' 2024-08-31T22:21:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=88, tag_tid=11, entry_tid=14668' 2024-08-31T22:21:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=88, tag_tid=11, entry_tid=14668' '!=' 'object_number=89, tag_tid=11, entry_tid=14581' 2024-08-31T22:21:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=88, tag_tid=11, entry_tid=14668' = 'object_number=89, tag_tid=11, entry_tid=14581' 2024-08-31T22:21:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=88, tag_tid=11, entry_tid=14668' 2024-08-31T22:21:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=88, tag_tid=11, entry_tid=14668' 2024-08-31T22:21:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=88, tag_tid=11, entry_tid=14668' 2024-08-31T22:21:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14668 2024-08-31T22:21:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 14668 2024-08-31T22:21:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:46.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:46.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=89, tag_tid=11, entry_tid=14765], [object_number=88, tag_tid=11, entry_tid=14764], [object_number=91, tag_tid=11, entry_tid=14763], [object_number=90, tag_tid=11, entry_tid=14762]]], state=connected] 2024-08-31T22:21:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:46.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:46.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:46.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=89, tag_tid=11, entry_tid=14765], [object_number=88, tag_tid=11, entry_tid=14764], [object_number=91, tag_tid=11, entry_tid=14763], [object_number=90, tag_tid=11, entry_tid=14762]]], state=connected] 2024-08-31T22:21:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=89, tag_tid=11, entry_tid=14765' 2024-08-31T22:21:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=89, tag_tid=11, entry_tid=14765' 2024-08-31T22:21:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=89, tag_tid=11, entry_tid=14765' '!=' 'object_number=88, tag_tid=11, entry_tid=14668' 2024-08-31T22:21:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=89, tag_tid=11, entry_tid=14765' = 'object_number=88, tag_tid=11, entry_tid=14668' 2024-08-31T22:21:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=89, tag_tid=11, entry_tid=14765' 2024-08-31T22:21:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=89, tag_tid=11, entry_tid=14765' 2024-08-31T22:21:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=89, tag_tid=11, entry_tid=14765' 2024-08-31T22:21:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14765 2024-08-31T22:21:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 14765 2024-08-31T22:21:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=14855], [object_number=90, tag_tid=11, entry_tid=14854], [object_number=89, tag_tid=11, entry_tid=14853], [object_number=88, tag_tid=11, entry_tid=14852]]], state=connected] 2024-08-31T22:21:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=14855], [object_number=90, tag_tid=11, entry_tid=14854], [object_number=89, tag_tid=11, entry_tid=14853], [object_number=88, tag_tid=11, entry_tid=14852]]], state=connected] 2024-08-31T22:21:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=91, tag_tid=11, entry_tid=14855' 2024-08-31T22:21:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=91, tag_tid=11, entry_tid=14855' 2024-08-31T22:21:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=14855' '!=' 'object_number=89, tag_tid=11, entry_tid=14765' 2024-08-31T22:21:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=14855' = 'object_number=89, tag_tid=11, entry_tid=14765' 2024-08-31T22:21:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=91, tag_tid=11, entry_tid=14855' 2024-08-31T22:21:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=14855' 2024-08-31T22:21:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=14855' 2024-08-31T22:21:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14855 2024-08-31T22:21:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 14855 2024-08-31T22:21:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:47.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:47.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:47.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:47.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=90, tag_tid=11, entry_tid=14950], [object_number=89, tag_tid=11, entry_tid=14949], [object_number=88, tag_tid=11, entry_tid=14948], [object_number=91, tag_tid=11, entry_tid=14947]]], state=connected] 2024-08-31T22:21:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=90, tag_tid=11, entry_tid=14950], [object_number=89, tag_tid=11, entry_tid=14949], [object_number=88, tag_tid=11, entry_tid=14948], [object_number=91, tag_tid=11, entry_tid=14947]]], state=connected] 2024-08-31T22:21:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=90, tag_tid=11, entry_tid=14950' 2024-08-31T22:21:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=90, tag_tid=11, entry_tid=14950' 2024-08-31T22:21:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=90, tag_tid=11, entry_tid=14950' '!=' 'object_number=91, tag_tid=11, entry_tid=14855' 2024-08-31T22:21:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=90, tag_tid=11, entry_tid=14950' = 'object_number=91, tag_tid=11, entry_tid=14855' 2024-08-31T22:21:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=90, tag_tid=11, entry_tid=14950' 2024-08-31T22:21:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=90, tag_tid=11, entry_tid=14950' 2024-08-31T22:21:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=90, tag_tid=11, entry_tid=14950' 2024-08-31T22:21:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14950 2024-08-31T22:21:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 14950 2024-08-31T22:21:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=88, tag_tid=11, entry_tid=15052], [object_number=91, tag_tid=11, entry_tid=15051], [object_number=90, tag_tid=11, entry_tid=15050], [object_number=89, tag_tid=11, entry_tid=15049]]], state=connected] 2024-08-31T22:21:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=88, tag_tid=11, entry_tid=15052], [object_number=91, tag_tid=11, entry_tid=15051], [object_number=90, tag_tid=11, entry_tid=15050], [object_number=89, tag_tid=11, entry_tid=15049]]], state=connected] 2024-08-31T22:21:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=88, tag_tid=11, entry_tid=15052' 2024-08-31T22:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=88, tag_tid=11, entry_tid=15052' 2024-08-31T22:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=88, tag_tid=11, entry_tid=15052' '!=' 'object_number=90, tag_tid=11, entry_tid=14950' 2024-08-31T22:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=88, tag_tid=11, entry_tid=15052' = 'object_number=90, tag_tid=11, entry_tid=14950' 2024-08-31T22:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=88, tag_tid=11, entry_tid=15052' 2024-08-31T22:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=88, tag_tid=11, entry_tid=15052' 2024-08-31T22:21:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=88, tag_tid=11, entry_tid=15052' 2024-08-31T22:21:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15052 2024-08-31T22:21:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 15052 2024-08-31T22:21:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=15155], [object_number=90, tag_tid=11, entry_tid=15154], [object_number=89, tag_tid=11, entry_tid=15153], [object_number=88, tag_tid=11, entry_tid=15152]]], state=connected] 2024-08-31T22:21:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=15155], [object_number=90, tag_tid=11, entry_tid=15154], [object_number=89, tag_tid=11, entry_tid=15153], [object_number=88, tag_tid=11, entry_tid=15152]]], state=connected] 2024-08-31T22:21:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=91, tag_tid=11, entry_tid=15155' 2024-08-31T22:21:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=91, tag_tid=11, entry_tid=15155' 2024-08-31T22:21:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=15155' '!=' 'object_number=88, tag_tid=11, entry_tid=15052' 2024-08-31T22:21:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=15155' = 'object_number=88, tag_tid=11, entry_tid=15052' 2024-08-31T22:21:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=91, tag_tid=11, entry_tid=15155' 2024-08-31T22:21:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=15155' 2024-08-31T22:21:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=15155' 2024-08-31T22:21:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15155 2024-08-31T22:21:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 15155 2024-08-31T22:21:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=15255], [object_number=90, tag_tid=11, entry_tid=15254], [object_number=89, tag_tid=11, entry_tid=15253], [object_number=88, tag_tid=11, entry_tid=15252]]], state=connected] 2024-08-31T22:21:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=15255], [object_number=90, tag_tid=11, entry_tid=15254], [object_number=89, tag_tid=11, entry_tid=15253], [object_number=88, tag_tid=11, entry_tid=15252]]], state=connected] 2024-08-31T22:21:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=91, tag_tid=11, entry_tid=15255' 2024-08-31T22:21:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=91, tag_tid=11, entry_tid=15255' 2024-08-31T22:21:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=15255' '!=' 'object_number=91, tag_tid=11, entry_tid=15155' 2024-08-31T22:21:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=15255' = 'object_number=91, tag_tid=11, entry_tid=15155' 2024-08-31T22:21:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=91, tag_tid=11, entry_tid=15255' 2024-08-31T22:21:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=15255' 2024-08-31T22:21:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=15255' 2024-08-31T22:21:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15255 2024-08-31T22:21:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 15255 2024-08-31T22:21:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=89, tag_tid=11, entry_tid=15401], [object_number=88, tag_tid=11, entry_tid=15400], [object_number=91, tag_tid=11, entry_tid=15399], [object_number=90, tag_tid=11, entry_tid=15398]]], state=connected] 2024-08-31T22:21:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=89, tag_tid=11, entry_tid=15401], [object_number=88, tag_tid=11, entry_tid=15400], [object_number=91, tag_tid=11, entry_tid=15399], [object_number=90, tag_tid=11, entry_tid=15398]]], state=connected] 2024-08-31T22:21:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=89, tag_tid=11, entry_tid=15401' 2024-08-31T22:21:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=89, tag_tid=11, entry_tid=15401' 2024-08-31T22:21:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=89, tag_tid=11, entry_tid=15401' '!=' 'object_number=91, tag_tid=11, entry_tid=15255' 2024-08-31T22:21:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=89, tag_tid=11, entry_tid=15401' = 'object_number=91, tag_tid=11, entry_tid=15255' 2024-08-31T22:21:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=89, tag_tid=11, entry_tid=15401' 2024-08-31T22:21:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=89, tag_tid=11, entry_tid=15401' 2024-08-31T22:21:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=89, tag_tid=11, entry_tid=15401' 2024-08-31T22:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15401 2024-08-31T22:21:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 15401 2024-08-31T22:21:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:50.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:50.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:50.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:50.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:50.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:50.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=88, tag_tid=11, entry_tid=15512], [object_number=91, tag_tid=11, entry_tid=15511], [object_number=90, tag_tid=11, entry_tid=15510], [object_number=89, tag_tid=11, entry_tid=15509]]], state=connected] 2024-08-31T22:21:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=88, tag_tid=11, entry_tid=15512], [object_number=91, tag_tid=11, entry_tid=15511], [object_number=90, tag_tid=11, entry_tid=15510], [object_number=89, tag_tid=11, entry_tid=15509]]], state=connected] 2024-08-31T22:21:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=88, tag_tid=11, entry_tid=15512' 2024-08-31T22:21:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=88, tag_tid=11, entry_tid=15512' 2024-08-31T22:21:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=88, tag_tid=11, entry_tid=15512' '!=' 'object_number=89, tag_tid=11, entry_tid=15401' 2024-08-31T22:21:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=88, tag_tid=11, entry_tid=15512' = 'object_number=89, tag_tid=11, entry_tid=15401' 2024-08-31T22:21:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=88, tag_tid=11, entry_tid=15512' 2024-08-31T22:21:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=88, tag_tid=11, entry_tid=15512' 2024-08-31T22:21:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=88, tag_tid=11, entry_tid=15512' 2024-08-31T22:21:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15512 2024-08-31T22:21:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 15512 2024-08-31T22:21:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=89, tag_tid=11, entry_tid=15617], [object_number=88, tag_tid=11, entry_tid=15616], [object_number=91, tag_tid=11, entry_tid=15615], [object_number=90, tag_tid=11, entry_tid=15614]]], state=connected] 2024-08-31T22:21:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=89, tag_tid=11, entry_tid=15617], [object_number=88, tag_tid=11, entry_tid=15616], [object_number=91, tag_tid=11, entry_tid=15615], [object_number=90, tag_tid=11, entry_tid=15614]]], state=connected] 2024-08-31T22:21:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=89, tag_tid=11, entry_tid=15617' 2024-08-31T22:21:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=89, tag_tid=11, entry_tid=15617' 2024-08-31T22:21:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=89, tag_tid=11, entry_tid=15617' '!=' 'object_number=88, tag_tid=11, entry_tid=15512' 2024-08-31T22:21:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=89, tag_tid=11, entry_tid=15617' = 'object_number=88, tag_tid=11, entry_tid=15512' 2024-08-31T22:21:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=89, tag_tid=11, entry_tid=15617' 2024-08-31T22:21:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=89, tag_tid=11, entry_tid=15617' 2024-08-31T22:21:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=89, tag_tid=11, entry_tid=15617' 2024-08-31T22:21:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15617 2024-08-31T22:21:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 15617 2024-08-31T22:21:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=15715], [object_number=90, tag_tid=11, entry_tid=15714], [object_number=89, tag_tid=11, entry_tid=15713], [object_number=88, tag_tid=11, entry_tid=15712]]], state=connected] 2024-08-31T22:21:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=15715], [object_number=90, tag_tid=11, entry_tid=15714], [object_number=89, tag_tid=11, entry_tid=15713], [object_number=88, tag_tid=11, entry_tid=15712]]], state=connected] 2024-08-31T22:21:51.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=91, tag_tid=11, entry_tid=15715' 2024-08-31T22:21:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=91, tag_tid=11, entry_tid=15715' 2024-08-31T22:21:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=15715' '!=' 'object_number=89, tag_tid=11, entry_tid=15617' 2024-08-31T22:21:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=15715' = 'object_number=89, tag_tid=11, entry_tid=15617' 2024-08-31T22:21:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=91, tag_tid=11, entry_tid=15715' 2024-08-31T22:21:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=15715' 2024-08-31T22:21:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:51.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:51.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:51.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=15715' 2024-08-31T22:21:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15715 2024-08-31T22:21:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 15715 2024-08-31T22:21:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=15803], [object_number=90, tag_tid=11, entry_tid=15802], [object_number=89, tag_tid=11, entry_tid=15801], [object_number=88, tag_tid=11, entry_tid=15800]]], state=connected] 2024-08-31T22:21:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:51.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:51.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:51.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:51.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:51.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:51.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:51.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:51.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=15803], [object_number=90, tag_tid=11, entry_tid=15802], [object_number=89, tag_tid=11, entry_tid=15801], [object_number=88, tag_tid=11, entry_tid=15800]]], state=connected] 2024-08-31T22:21:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=91, tag_tid=11, entry_tid=15803' 2024-08-31T22:21:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=91, tag_tid=11, entry_tid=15803' 2024-08-31T22:21:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=15803' '!=' 'object_number=91, tag_tid=11, entry_tid=15715' 2024-08-31T22:21:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=15803' = 'object_number=91, tag_tid=11, entry_tid=15715' 2024-08-31T22:21:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=91, tag_tid=11, entry_tid=15803' 2024-08-31T22:21:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=15803' 2024-08-31T22:21:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:51.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:51.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=15803' 2024-08-31T22:21:51.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:51.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15803 2024-08-31T22:21:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 15803 2024-08-31T22:21:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:51.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:51.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=15879], [object_number=90, tag_tid=11, entry_tid=15878], [object_number=89, tag_tid=11, entry_tid=15877], [object_number=88, tag_tid=11, entry_tid=15876]]], state=connected] 2024-08-31T22:21:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:52.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:52.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:52.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:52.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=91, tag_tid=11, entry_tid=15879], [object_number=90, tag_tid=11, entry_tid=15878], [object_number=89, tag_tid=11, entry_tid=15877], [object_number=88, tag_tid=11, entry_tid=15876]]], state=connected] 2024-08-31T22:21:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=91, tag_tid=11, entry_tid=15879' 2024-08-31T22:21:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=91, tag_tid=11, entry_tid=15879' 2024-08-31T22:21:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=15879' '!=' 'object_number=91, tag_tid=11, entry_tid=15803' 2024-08-31T22:21:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=91, tag_tid=11, entry_tid=15879' = 'object_number=91, tag_tid=11, entry_tid=15803' 2024-08-31T22:21:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=91, tag_tid=11, entry_tid=15879' 2024-08-31T22:21:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=15879' 2024-08-31T22:21:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=91, tag_tid=11, entry_tid=15879' 2024-08-31T22:21:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15879 2024-08-31T22:21:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 15879 2024-08-31T22:21:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=90, tag_tid=11, entry_tid=15958], [object_number=89, tag_tid=11, entry_tid=15957], [object_number=88, tag_tid=11, entry_tid=15956], [object_number=91, tag_tid=11, entry_tid=15955]]], state=connected] 2024-08-31T22:21:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=90, tag_tid=11, entry_tid=15958], [object_number=89, tag_tid=11, entry_tid=15957], [object_number=88, tag_tid=11, entry_tid=15956], [object_number=91, tag_tid=11, entry_tid=15955]]], state=connected] 2024-08-31T22:21:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=90, tag_tid=11, entry_tid=15958' 2024-08-31T22:21:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=90, tag_tid=11, entry_tid=15958' 2024-08-31T22:21:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=90, tag_tid=11, entry_tid=15958' '!=' 'object_number=91, tag_tid=11, entry_tid=15879' 2024-08-31T22:21:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=90, tag_tid=11, entry_tid=15958' = 'object_number=91, tag_tid=11, entry_tid=15879' 2024-08-31T22:21:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=90, tag_tid=11, entry_tid=15958' 2024-08-31T22:21:52.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:52.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:52.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:52.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=90, tag_tid=11, entry_tid=15958' 2024-08-31T22:21:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=90, tag_tid=11, entry_tid=15958' 2024-08-31T22:21:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15958 2024-08-31T22:21:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 15958 2024-08-31T22:21:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:53.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=89, tag_tid=11, entry_tid=16021], [object_number=88, tag_tid=11, entry_tid=16020], [object_number=91, tag_tid=11, entry_tid=16019], [object_number=90, tag_tid=11, entry_tid=16018]]], state=connected] 2024-08-31T22:21:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 22 2024-08-31T22:21:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=89, tag_tid=11, entry_tid=16021], [object_number=88, tag_tid=11, entry_tid=16020], [object_number=91, tag_tid=11, entry_tid=16019], [object_number=90, tag_tid=11, entry_tid=16018]]], state=connected] 2024-08-31T22:21:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=89, tag_tid=11, entry_tid=16021' 2024-08-31T22:21:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=89, tag_tid=11, entry_tid=16021' 2024-08-31T22:21:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=89, tag_tid=11, entry_tid=16021' '!=' 'object_number=90, tag_tid=11, entry_tid=15958' 2024-08-31T22:21:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=89, tag_tid=11, entry_tid=16021' = 'object_number=90, tag_tid=11, entry_tid=15958' 2024-08-31T22:21:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=89, tag_tid=11, entry_tid=16021' 2024-08-31T22:21:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=89, tag_tid=11, entry_tid=16021' 2024-08-31T22:21:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=89, tag_tid=11, entry_tid=16021' 2024-08-31T22:21:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16021 2024-08-31T22:21:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 16021 2024-08-31T22:21:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=16069], [object_number=92, tag_tid=11, entry_tid=16068], [object_number=95, tag_tid=11, entry_tid=16067], [object_number=94, tag_tid=11, entry_tid=16066]]], state=connected] 2024-08-31T22:21:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=16069], [object_number=92, tag_tid=11, entry_tid=16068], [object_number=95, tag_tid=11, entry_tid=16067], [object_number=94, tag_tid=11, entry_tid=16066]]], state=connected] 2024-08-31T22:21:54.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=93, tag_tid=11, entry_tid=16069' 2024-08-31T22:21:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=93, tag_tid=11, entry_tid=16069' 2024-08-31T22:21:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=16069' '!=' 'object_number=89, tag_tid=11, entry_tid=16021' 2024-08-31T22:21:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=16069' = 'object_number=89, tag_tid=11, entry_tid=16021' 2024-08-31T22:21:54.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=93, tag_tid=11, entry_tid=16069' 2024-08-31T22:21:54.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:54.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:54.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:54.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:54.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=16069' 2024-08-31T22:21:54.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:54.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=16069' 2024-08-31T22:21:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16069 2024-08-31T22:21:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 16069 2024-08-31T22:21:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=16173], [object_number=92, tag_tid=11, entry_tid=16172], [object_number=95, tag_tid=11, entry_tid=16171], [object_number=94, tag_tid=11, entry_tid=16170]]], state=connected] 2024-08-31T22:21:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:54.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=16173], [object_number=92, tag_tid=11, entry_tid=16172], [object_number=95, tag_tid=11, entry_tid=16171], [object_number=94, tag_tid=11, entry_tid=16170]]], state=connected] 2024-08-31T22:21:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=93, tag_tid=11, entry_tid=16173' 2024-08-31T22:21:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=93, tag_tid=11, entry_tid=16173' 2024-08-31T22:21:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=16173' '!=' 'object_number=93, tag_tid=11, entry_tid=16069' 2024-08-31T22:21:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=16173' = 'object_number=93, tag_tid=11, entry_tid=16069' 2024-08-31T22:21:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=93, tag_tid=11, entry_tid=16173' 2024-08-31T22:21:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=16173' 2024-08-31T22:21:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:54.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=16173' 2024-08-31T22:21:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16173 2024-08-31T22:21:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 16173 2024-08-31T22:21:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:54.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:54.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:54.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:54.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=16281], [object_number=92, tag_tid=11, entry_tid=16280], [object_number=95, tag_tid=11, entry_tid=16279], [object_number=94, tag_tid=11, entry_tid=16278]]], state=connected] 2024-08-31T22:21:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=16281], [object_number=92, tag_tid=11, entry_tid=16280], [object_number=95, tag_tid=11, entry_tid=16279], [object_number=94, tag_tid=11, entry_tid=16278]]], state=connected] 2024-08-31T22:21:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=93, tag_tid=11, entry_tid=16281' 2024-08-31T22:21:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=93, tag_tid=11, entry_tid=16281' 2024-08-31T22:21:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=16281' '!=' 'object_number=93, tag_tid=11, entry_tid=16173' 2024-08-31T22:21:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=16281' = 'object_number=93, tag_tid=11, entry_tid=16173' 2024-08-31T22:21:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=93, tag_tid=11, entry_tid=16281' 2024-08-31T22:21:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=16281' 2024-08-31T22:21:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=16281' 2024-08-31T22:21:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16281 2024-08-31T22:21:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 16281 2024-08-31T22:21:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=95, tag_tid=11, entry_tid=16439], [object_number=94, tag_tid=11, entry_tid=16438], [object_number=93, tag_tid=11, entry_tid=16437], [object_number=92, tag_tid=11, entry_tid=16436]]], state=connected] 2024-08-31T22:21:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=95, tag_tid=11, entry_tid=16439], [object_number=94, tag_tid=11, entry_tid=16438], [object_number=93, tag_tid=11, entry_tid=16437], [object_number=92, tag_tid=11, entry_tid=16436]]], state=connected] 2024-08-31T22:21:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=95, tag_tid=11, entry_tid=16439' 2024-08-31T22:21:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=95, tag_tid=11, entry_tid=16439' 2024-08-31T22:21:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=95, tag_tid=11, entry_tid=16439' '!=' 'object_number=93, tag_tid=11, entry_tid=16281' 2024-08-31T22:21:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=95, tag_tid=11, entry_tid=16439' = 'object_number=93, tag_tid=11, entry_tid=16281' 2024-08-31T22:21:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=95, tag_tid=11, entry_tid=16439' 2024-08-31T22:21:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:55.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=95, tag_tid=11, entry_tid=16439' 2024-08-31T22:21:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=95, tag_tid=11, entry_tid=16439' 2024-08-31T22:21:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16439 2024-08-31T22:21:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 16439 2024-08-31T22:21:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=92, tag_tid=11, entry_tid=16544], [object_number=95, tag_tid=11, entry_tid=16543], [object_number=94, tag_tid=11, entry_tid=16542], [object_number=93, tag_tid=11, entry_tid=16541]]], state=connected] 2024-08-31T22:21:56.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:56.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=92, tag_tid=11, entry_tid=16544], [object_number=95, tag_tid=11, entry_tid=16543], [object_number=94, tag_tid=11, entry_tid=16542], [object_number=93, tag_tid=11, entry_tid=16541]]], state=connected] 2024-08-31T22:21:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=92, tag_tid=11, entry_tid=16544' 2024-08-31T22:21:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=92, tag_tid=11, entry_tid=16544' 2024-08-31T22:21:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=92, tag_tid=11, entry_tid=16544' '!=' 'object_number=95, tag_tid=11, entry_tid=16439' 2024-08-31T22:21:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=92, tag_tid=11, entry_tid=16544' = 'object_number=95, tag_tid=11, entry_tid=16439' 2024-08-31T22:21:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=92, tag_tid=11, entry_tid=16544' 2024-08-31T22:21:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=92, tag_tid=11, entry_tid=16544' 2024-08-31T22:21:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=92, tag_tid=11, entry_tid=16544' 2024-08-31T22:21:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16544 2024-08-31T22:21:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 16544 2024-08-31T22:21:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=94, tag_tid=11, entry_tid=16646], [object_number=93, tag_tid=11, entry_tid=16645], [object_number=92, tag_tid=11, entry_tid=16644], [object_number=95, tag_tid=11, entry_tid=16643]]], state=connected] 2024-08-31T22:21:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=94, tag_tid=11, entry_tid=16646], [object_number=93, tag_tid=11, entry_tid=16645], [object_number=92, tag_tid=11, entry_tid=16644], [object_number=95, tag_tid=11, entry_tid=16643]]], state=connected] 2024-08-31T22:21:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=94, tag_tid=11, entry_tid=16646' 2024-08-31T22:21:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=94, tag_tid=11, entry_tid=16646' 2024-08-31T22:21:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=94, tag_tid=11, entry_tid=16646' '!=' 'object_number=92, tag_tid=11, entry_tid=16544' 2024-08-31T22:21:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=94, tag_tid=11, entry_tid=16646' = 'object_number=92, tag_tid=11, entry_tid=16544' 2024-08-31T22:21:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=94, tag_tid=11, entry_tid=16646' 2024-08-31T22:21:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=94, tag_tid=11, entry_tid=16646' 2024-08-31T22:21:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=94, tag_tid=11, entry_tid=16646' 2024-08-31T22:21:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16646 2024-08-31T22:21:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 16646 2024-08-31T22:21:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:56.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=16757], [object_number=92, tag_tid=11, entry_tid=16756], [object_number=95, tag_tid=11, entry_tid=16755], [object_number=94, tag_tid=11, entry_tid=16754]]], state=connected] 2024-08-31T22:21:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:57.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:57.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=16757], [object_number=92, tag_tid=11, entry_tid=16756], [object_number=95, tag_tid=11, entry_tid=16755], [object_number=94, tag_tid=11, entry_tid=16754]]], state=connected] 2024-08-31T22:21:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=93, tag_tid=11, entry_tid=16757' 2024-08-31T22:21:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=93, tag_tid=11, entry_tid=16757' 2024-08-31T22:21:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=16757' '!=' 'object_number=94, tag_tid=11, entry_tid=16646' 2024-08-31T22:21:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=16757' = 'object_number=94, tag_tid=11, entry_tid=16646' 2024-08-31T22:21:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=93, tag_tid=11, entry_tid=16757' 2024-08-31T22:21:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=16757' 2024-08-31T22:21:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:57.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=16757' 2024-08-31T22:21:57.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:57.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16757 2024-08-31T22:21:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 16757 2024-08-31T22:21:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:57.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:57.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:57.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:57.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:57.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:57.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:57.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:57.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=92, tag_tid=11, entry_tid=16868], [object_number=95, tag_tid=11, entry_tid=16867], [object_number=94, tag_tid=11, entry_tid=16866], [object_number=93, tag_tid=11, entry_tid=16865]]], state=connected] 2024-08-31T22:21:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=92, tag_tid=11, entry_tid=16868], [object_number=95, tag_tid=11, entry_tid=16867], [object_number=94, tag_tid=11, entry_tid=16866], [object_number=93, tag_tid=11, entry_tid=16865]]], state=connected] 2024-08-31T22:21:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:57.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=92, tag_tid=11, entry_tid=16868' 2024-08-31T22:21:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=92, tag_tid=11, entry_tid=16868' 2024-08-31T22:21:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=92, tag_tid=11, entry_tid=16868' '!=' 'object_number=93, tag_tid=11, entry_tid=16757' 2024-08-31T22:21:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=92, tag_tid=11, entry_tid=16868' = 'object_number=93, tag_tid=11, entry_tid=16757' 2024-08-31T22:21:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=92, tag_tid=11, entry_tid=16868' 2024-08-31T22:21:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:57.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:57.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=92, tag_tid=11, entry_tid=16868' 2024-08-31T22:21:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:57.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=92, tag_tid=11, entry_tid=16868' 2024-08-31T22:21:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16868 2024-08-31T22:21:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 16868 2024-08-31T22:21:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:58.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:58.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:58.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=94, tag_tid=11, entry_tid=17018], [object_number=93, tag_tid=11, entry_tid=17017], [object_number=92, tag_tid=11, entry_tid=17016], [object_number=95, tag_tid=11, entry_tid=17015]]], state=connected] 2024-08-31T22:21:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=94, tag_tid=11, entry_tid=17018], [object_number=93, tag_tid=11, entry_tid=17017], [object_number=92, tag_tid=11, entry_tid=17016], [object_number=95, tag_tid=11, entry_tid=17015]]], state=connected] 2024-08-31T22:21:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=94, tag_tid=11, entry_tid=17018' 2024-08-31T22:21:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=94, tag_tid=11, entry_tid=17018' 2024-08-31T22:21:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=94, tag_tid=11, entry_tid=17018' '!=' 'object_number=92, tag_tid=11, entry_tid=16868' 2024-08-31T22:21:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=94, tag_tid=11, entry_tid=17018' = 'object_number=92, tag_tid=11, entry_tid=16868' 2024-08-31T22:21:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=94, tag_tid=11, entry_tid=17018' 2024-08-31T22:21:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=94, tag_tid=11, entry_tid=17018' 2024-08-31T22:21:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=94, tag_tid=11, entry_tid=17018' 2024-08-31T22:21:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17018 2024-08-31T22:21:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 17018 2024-08-31T22:21:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=92, tag_tid=11, entry_tid=17124], [object_number=95, tag_tid=11, entry_tid=17123], [object_number=94, tag_tid=11, entry_tid=17122], [object_number=93, tag_tid=11, entry_tid=17121]]], state=connected] 2024-08-31T22:21:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=92, tag_tid=11, entry_tid=17124], [object_number=95, tag_tid=11, entry_tid=17123], [object_number=94, tag_tid=11, entry_tid=17122], [object_number=93, tag_tid=11, entry_tid=17121]]], state=connected] 2024-08-31T22:21:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=92, tag_tid=11, entry_tid=17124' 2024-08-31T22:21:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=92, tag_tid=11, entry_tid=17124' 2024-08-31T22:21:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=92, tag_tid=11, entry_tid=17124' '!=' 'object_number=94, tag_tid=11, entry_tid=17018' 2024-08-31T22:21:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=92, tag_tid=11, entry_tid=17124' = 'object_number=94, tag_tid=11, entry_tid=17018' 2024-08-31T22:21:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=92, tag_tid=11, entry_tid=17124' 2024-08-31T22:21:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=92, tag_tid=11, entry_tid=17124' 2024-08-31T22:21:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=92, tag_tid=11, entry_tid=17124' 2024-08-31T22:21:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17124 2024-08-31T22:21:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 17124 2024-08-31T22:21:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:59.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:59.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:59.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:59.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:59.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:59.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=95, tag_tid=11, entry_tid=17215], [object_number=94, tag_tid=11, entry_tid=17214], [object_number=93, tag_tid=11, entry_tid=17213], [object_number=92, tag_tid=11, entry_tid=17212]]], state=connected] 2024-08-31T22:21:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=95, tag_tid=11, entry_tid=17215], [object_number=94, tag_tid=11, entry_tid=17214], [object_number=93, tag_tid=11, entry_tid=17213], [object_number=92, tag_tid=11, entry_tid=17212]]], state=connected] 2024-08-31T22:21:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=95, tag_tid=11, entry_tid=17215' 2024-08-31T22:21:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=95, tag_tid=11, entry_tid=17215' 2024-08-31T22:21:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=95, tag_tid=11, entry_tid=17215' '!=' 'object_number=92, tag_tid=11, entry_tid=17124' 2024-08-31T22:21:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=95, tag_tid=11, entry_tid=17215' = 'object_number=92, tag_tid=11, entry_tid=17124' 2024-08-31T22:21:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=95, tag_tid=11, entry_tid=17215' 2024-08-31T22:21:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=95, tag_tid=11, entry_tid=17215' 2024-08-31T22:21:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=95, tag_tid=11, entry_tid=17215' 2024-08-31T22:21:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17215 2024-08-31T22:21:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 17215 2024-08-31T22:21:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:21:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:21:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:21:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:21:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:21:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:21:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:21:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:21:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:21:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:21:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:21:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:21:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:21:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:21:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:21:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:21:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:21:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:21:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:21:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:21:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:21:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:21:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:21:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:21:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:21:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:21:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:21:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:21:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:21:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:21:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:21:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:21:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:21:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:21:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:21:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:21:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:21:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:21:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:21:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:21:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:21:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:21:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:21:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:21:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:21:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:21:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:21:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:21:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:21:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:21:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:21:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=92, tag_tid=11, entry_tid=17304], [object_number=95, tag_tid=11, entry_tid=17303], [object_number=94, tag_tid=11, entry_tid=17302], [object_number=93, tag_tid=11, entry_tid=17301]]], state=connected] 2024-08-31T22:21:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:21:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:21:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:21:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:21:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:21:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:21:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:21:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:21:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:21:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:21:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:21:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:21:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:21:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:21:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=92, tag_tid=11, entry_tid=17304], [object_number=95, tag_tid=11, entry_tid=17303], [object_number=94, tag_tid=11, entry_tid=17302], [object_number=93, tag_tid=11, entry_tid=17301]]], state=connected] 2024-08-31T22:21:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:21:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=92, tag_tid=11, entry_tid=17304' 2024-08-31T22:21:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=92, tag_tid=11, entry_tid=17304' 2024-08-31T22:21:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=92, tag_tid=11, entry_tid=17304' '!=' 'object_number=95, tag_tid=11, entry_tid=17215' 2024-08-31T22:21:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:21:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=92, tag_tid=11, entry_tid=17304' = 'object_number=95, tag_tid=11, entry_tid=17215' 2024-08-31T22:21:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=92, tag_tid=11, entry_tid=17304' 2024-08-31T22:21:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:21:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=92, tag_tid=11, entry_tid=17304' 2024-08-31T22:21:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:21:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:21:59.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:21:59.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:59.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:21:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=92, tag_tid=11, entry_tid=17304' 2024-08-31T22:21:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:21:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:21:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17304 2024-08-31T22:21:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 17304 2024-08-31T22:21:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:21:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:21:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:00.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:00.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:00.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:00.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:00.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=17397], [object_number=92, tag_tid=11, entry_tid=17396], [object_number=95, tag_tid=11, entry_tid=17395], [object_number=94, tag_tid=11, entry_tid=17394]]], state=connected] 2024-08-31T22:22:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=17397], [object_number=92, tag_tid=11, entry_tid=17396], [object_number=95, tag_tid=11, entry_tid=17395], [object_number=94, tag_tid=11, entry_tid=17394]]], state=connected] 2024-08-31T22:22:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=93, tag_tid=11, entry_tid=17397' 2024-08-31T22:22:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=93, tag_tid=11, entry_tid=17397' 2024-08-31T22:22:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=17397' '!=' 'object_number=92, tag_tid=11, entry_tid=17304' 2024-08-31T22:22:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=17397' = 'object_number=92, tag_tid=11, entry_tid=17304' 2024-08-31T22:22:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=93, tag_tid=11, entry_tid=17397' 2024-08-31T22:22:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=17397' 2024-08-31T22:22:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=17397' 2024-08-31T22:22:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17397 2024-08-31T22:22:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 17397 2024-08-31T22:22:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:00.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:00.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:00.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:00.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:00.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=95, tag_tid=11, entry_tid=17487], [object_number=94, tag_tid=11, entry_tid=17486], [object_number=93, tag_tid=11, entry_tid=17485], [object_number=92, tag_tid=11, entry_tid=17484]]], state=connected] 2024-08-31T22:22:00.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=95, tag_tid=11, entry_tid=17487], [object_number=94, tag_tid=11, entry_tid=17486], [object_number=93, tag_tid=11, entry_tid=17485], [object_number=92, tag_tid=11, entry_tid=17484]]], state=connected] 2024-08-31T22:22:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=95, tag_tid=11, entry_tid=17487' 2024-08-31T22:22:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=95, tag_tid=11, entry_tid=17487' 2024-08-31T22:22:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=95, tag_tid=11, entry_tid=17487' '!=' 'object_number=93, tag_tid=11, entry_tid=17397' 2024-08-31T22:22:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=95, tag_tid=11, entry_tid=17487' = 'object_number=93, tag_tid=11, entry_tid=17397' 2024-08-31T22:22:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=95, tag_tid=11, entry_tid=17487' 2024-08-31T22:22:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=95, tag_tid=11, entry_tid=17487' 2024-08-31T22:22:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=95, tag_tid=11, entry_tid=17487' 2024-08-31T22:22:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17487 2024-08-31T22:22:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 17487 2024-08-31T22:22:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:01.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:01.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:01.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:01.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=94, tag_tid=11, entry_tid=17582], [object_number=93, tag_tid=11, entry_tid=17581], [object_number=92, tag_tid=11, entry_tid=17580], [object_number=95, tag_tid=11, entry_tid=17579]]], state=connected] 2024-08-31T22:22:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:01.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:01.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:01.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:01.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=94, tag_tid=11, entry_tid=17582], [object_number=93, tag_tid=11, entry_tid=17581], [object_number=92, tag_tid=11, entry_tid=17580], [object_number=95, tag_tid=11, entry_tid=17579]]], state=connected] 2024-08-31T22:22:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:01.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=94, tag_tid=11, entry_tid=17582' 2024-08-31T22:22:01.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=94, tag_tid=11, entry_tid=17582' 2024-08-31T22:22:01.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=94, tag_tid=11, entry_tid=17582' '!=' 'object_number=95, tag_tid=11, entry_tid=17487' 2024-08-31T22:22:01.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:01.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=94, tag_tid=11, entry_tid=17582' = 'object_number=95, tag_tid=11, entry_tid=17487' 2024-08-31T22:22:01.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=94, tag_tid=11, entry_tid=17582' 2024-08-31T22:22:01.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:01.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:01.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=94, tag_tid=11, entry_tid=17582' 2024-08-31T22:22:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=94, tag_tid=11, entry_tid=17582' 2024-08-31T22:22:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17582 2024-08-31T22:22:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 17582 2024-08-31T22:22:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:01.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:01.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:01.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:01.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:01.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:01.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:01.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=95, tag_tid=11, entry_tid=17671], [object_number=94, tag_tid=11, entry_tid=17670], [object_number=93, tag_tid=11, entry_tid=17669], [object_number=92, tag_tid=11, entry_tid=17668]]], state=connected] 2024-08-31T22:22:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=95, tag_tid=11, entry_tid=17671], [object_number=94, tag_tid=11, entry_tid=17670], [object_number=93, tag_tid=11, entry_tid=17669], [object_number=92, tag_tid=11, entry_tid=17668]]], state=connected] 2024-08-31T22:22:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=95, tag_tid=11, entry_tid=17671' 2024-08-31T22:22:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=95, tag_tid=11, entry_tid=17671' 2024-08-31T22:22:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=95, tag_tid=11, entry_tid=17671' '!=' 'object_number=94, tag_tid=11, entry_tid=17582' 2024-08-31T22:22:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=95, tag_tid=11, entry_tid=17671' = 'object_number=94, tag_tid=11, entry_tid=17582' 2024-08-31T22:22:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=95, tag_tid=11, entry_tid=17671' 2024-08-31T22:22:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=95, tag_tid=11, entry_tid=17671' 2024-08-31T22:22:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=95, tag_tid=11, entry_tid=17671' 2024-08-31T22:22:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17671 2024-08-31T22:22:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 17671 2024-08-31T22:22:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:02.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:02.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:02.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:02.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:02.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=17773], [object_number=92, tag_tid=11, entry_tid=17772], [object_number=95, tag_tid=11, entry_tid=17771], [object_number=94, tag_tid=11, entry_tid=17770]]], state=connected] 2024-08-31T22:22:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=17773], [object_number=92, tag_tid=11, entry_tid=17772], [object_number=95, tag_tid=11, entry_tid=17771], [object_number=94, tag_tid=11, entry_tid=17770]]], state=connected] 2024-08-31T22:22:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=93, tag_tid=11, entry_tid=17773' 2024-08-31T22:22:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=93, tag_tid=11, entry_tid=17773' 2024-08-31T22:22:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=17773' '!=' 'object_number=95, tag_tid=11, entry_tid=17671' 2024-08-31T22:22:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=17773' = 'object_number=95, tag_tid=11, entry_tid=17671' 2024-08-31T22:22:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=93, tag_tid=11, entry_tid=17773' 2024-08-31T22:22:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=17773' 2024-08-31T22:22:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=17773' 2024-08-31T22:22:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17773 2024-08-31T22:22:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 17773 2024-08-31T22:22:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:02.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:02.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:02.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:02.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:02.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:02.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=17873], [object_number=92, tag_tid=11, entry_tid=17872], [object_number=95, tag_tid=11, entry_tid=17871], [object_number=94, tag_tid=11, entry_tid=17870]]], state=connected] 2024-08-31T22:22:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:02.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:02.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:02.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:02.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:02.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=17873], [object_number=92, tag_tid=11, entry_tid=17872], [object_number=95, tag_tid=11, entry_tid=17871], [object_number=94, tag_tid=11, entry_tid=17870]]], state=connected] 2024-08-31T22:22:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=93, tag_tid=11, entry_tid=17873' 2024-08-31T22:22:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=93, tag_tid=11, entry_tid=17873' 2024-08-31T22:22:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=17873' '!=' 'object_number=93, tag_tid=11, entry_tid=17773' 2024-08-31T22:22:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=17873' = 'object_number=93, tag_tid=11, entry_tid=17773' 2024-08-31T22:22:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=93, tag_tid=11, entry_tid=17873' 2024-08-31T22:22:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:02.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:02.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=17873' 2024-08-31T22:22:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=17873' 2024-08-31T22:22:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17873 2024-08-31T22:22:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 17873 2024-08-31T22:22:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:03.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:03.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=92, tag_tid=11, entry_tid=17972], [object_number=95, tag_tid=11, entry_tid=17971], [object_number=94, tag_tid=11, entry_tid=17970], [object_number=93, tag_tid=11, entry_tid=17969]]], state=connected] 2024-08-31T22:22:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=92, tag_tid=11, entry_tid=17972], [object_number=95, tag_tid=11, entry_tid=17971], [object_number=94, tag_tid=11, entry_tid=17970], [object_number=93, tag_tid=11, entry_tid=17969]]], state=connected] 2024-08-31T22:22:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=92, tag_tid=11, entry_tid=17972' 2024-08-31T22:22:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=92, tag_tid=11, entry_tid=17972' 2024-08-31T22:22:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=92, tag_tid=11, entry_tid=17972' '!=' 'object_number=93, tag_tid=11, entry_tid=17873' 2024-08-31T22:22:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=92, tag_tid=11, entry_tid=17972' = 'object_number=93, tag_tid=11, entry_tid=17873' 2024-08-31T22:22:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=92, tag_tid=11, entry_tid=17972' 2024-08-31T22:22:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=92, tag_tid=11, entry_tid=17972' 2024-08-31T22:22:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=92, tag_tid=11, entry_tid=17972' 2024-08-31T22:22:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17972 2024-08-31T22:22:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 17972 2024-08-31T22:22:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:03.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:03.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:03.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=95, tag_tid=11, entry_tid=18067], [object_number=94, tag_tid=11, entry_tid=18066], [object_number=93, tag_tid=11, entry_tid=18065], [object_number=92, tag_tid=11, entry_tid=18064]]], state=connected] 2024-08-31T22:22:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=95, tag_tid=11, entry_tid=18067], [object_number=94, tag_tid=11, entry_tid=18066], [object_number=93, tag_tid=11, entry_tid=18065], [object_number=92, tag_tid=11, entry_tid=18064]]], state=connected] 2024-08-31T22:22:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=95, tag_tid=11, entry_tid=18067' 2024-08-31T22:22:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=95, tag_tid=11, entry_tid=18067' 2024-08-31T22:22:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=95, tag_tid=11, entry_tid=18067' '!=' 'object_number=92, tag_tid=11, entry_tid=17972' 2024-08-31T22:22:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=95, tag_tid=11, entry_tid=18067' = 'object_number=92, tag_tid=11, entry_tid=17972' 2024-08-31T22:22:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=95, tag_tid=11, entry_tid=18067' 2024-08-31T22:22:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=95, tag_tid=11, entry_tid=18067' 2024-08-31T22:22:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:03.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:03.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:03.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:03.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:03.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:03.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=95, tag_tid=11, entry_tid=18067' 2024-08-31T22:22:03.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:03.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18067 2024-08-31T22:22:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 18067 2024-08-31T22:22:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:04.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:04.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=18153], [object_number=92, tag_tid=11, entry_tid=18152], [object_number=95, tag_tid=11, entry_tid=18151], [object_number=94, tag_tid=11, entry_tid=18150]]], state=connected] 2024-08-31T22:22:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:04.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=18153], [object_number=92, tag_tid=11, entry_tid=18152], [object_number=95, tag_tid=11, entry_tid=18151], [object_number=94, tag_tid=11, entry_tid=18150]]], state=connected] 2024-08-31T22:22:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=93, tag_tid=11, entry_tid=18153' 2024-08-31T22:22:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=93, tag_tid=11, entry_tid=18153' 2024-08-31T22:22:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=18153' '!=' 'object_number=95, tag_tid=11, entry_tid=18067' 2024-08-31T22:22:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=18153' = 'object_number=95, tag_tid=11, entry_tid=18067' 2024-08-31T22:22:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=93, tag_tid=11, entry_tid=18153' 2024-08-31T22:22:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=18153' 2024-08-31T22:22:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=18153' 2024-08-31T22:22:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18153 2024-08-31T22:22:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 18153 2024-08-31T22:22:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=95, tag_tid=11, entry_tid=18239], [object_number=94, tag_tid=11, entry_tid=18238], [object_number=93, tag_tid=11, entry_tid=18237], [object_number=92, tag_tid=11, entry_tid=18236]]], state=connected] 2024-08-31T22:22:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=95, tag_tid=11, entry_tid=18239], [object_number=94, tag_tid=11, entry_tid=18238], [object_number=93, tag_tid=11, entry_tid=18237], [object_number=92, tag_tid=11, entry_tid=18236]]], state=connected] 2024-08-31T22:22:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=95, tag_tid=11, entry_tid=18239' 2024-08-31T22:22:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=95, tag_tid=11, entry_tid=18239' 2024-08-31T22:22:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=95, tag_tid=11, entry_tid=18239' '!=' 'object_number=93, tag_tid=11, entry_tid=18153' 2024-08-31T22:22:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=95, tag_tid=11, entry_tid=18239' = 'object_number=93, tag_tid=11, entry_tid=18153' 2024-08-31T22:22:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=95, tag_tid=11, entry_tid=18239' 2024-08-31T22:22:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=95, tag_tid=11, entry_tid=18239' 2024-08-31T22:22:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=95, tag_tid=11, entry_tid=18239' 2024-08-31T22:22:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18239 2024-08-31T22:22:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 18239 2024-08-31T22:22:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=95, tag_tid=11, entry_tid=18339], [object_number=94, tag_tid=11, entry_tid=18338], [object_number=93, tag_tid=11, entry_tid=18337], [object_number=92, tag_tid=11, entry_tid=18336]]], state=connected] 2024-08-31T22:22:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:05.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:05.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:05.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:05.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:05.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=95, tag_tid=11, entry_tid=18339], [object_number=94, tag_tid=11, entry_tid=18338], [object_number=93, tag_tid=11, entry_tid=18337], [object_number=92, tag_tid=11, entry_tid=18336]]], state=connected] 2024-08-31T22:22:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=95, tag_tid=11, entry_tid=18339' 2024-08-31T22:22:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=95, tag_tid=11, entry_tid=18339' 2024-08-31T22:22:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=95, tag_tid=11, entry_tid=18339' '!=' 'object_number=95, tag_tid=11, entry_tid=18239' 2024-08-31T22:22:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=95, tag_tid=11, entry_tid=18339' = 'object_number=95, tag_tid=11, entry_tid=18239' 2024-08-31T22:22:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=95, tag_tid=11, entry_tid=18339' 2024-08-31T22:22:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=95, tag_tid=11, entry_tid=18339' 2024-08-31T22:22:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=95, tag_tid=11, entry_tid=18339' 2024-08-31T22:22:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18339 2024-08-31T22:22:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 18339 2024-08-31T22:22:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=18453], [object_number=92, tag_tid=11, entry_tid=18452], [object_number=95, tag_tid=11, entry_tid=18451], [object_number=94, tag_tid=11, entry_tid=18450]]], state=connected] 2024-08-31T22:22:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:05.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:05.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:05.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=18453], [object_number=92, tag_tid=11, entry_tid=18452], [object_number=95, tag_tid=11, entry_tid=18451], [object_number=94, tag_tid=11, entry_tid=18450]]], state=connected] 2024-08-31T22:22:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=93, tag_tid=11, entry_tid=18453' 2024-08-31T22:22:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=93, tag_tid=11, entry_tid=18453' 2024-08-31T22:22:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=18453' '!=' 'object_number=95, tag_tid=11, entry_tid=18339' 2024-08-31T22:22:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=18453' = 'object_number=95, tag_tid=11, entry_tid=18339' 2024-08-31T22:22:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=93, tag_tid=11, entry_tid=18453' 2024-08-31T22:22:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=18453' 2024-08-31T22:22:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=18453' 2024-08-31T22:22:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18453 2024-08-31T22:22:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 18453 2024-08-31T22:22:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:06.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:06.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:06.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=92, tag_tid=11, entry_tid=18568], [object_number=95, tag_tid=11, entry_tid=18567], [object_number=94, tag_tid=11, entry_tid=18566], [object_number=93, tag_tid=11, entry_tid=18565]]], state=connected] 2024-08-31T22:22:06.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=92, tag_tid=11, entry_tid=18568], [object_number=95, tag_tid=11, entry_tid=18567], [object_number=94, tag_tid=11, entry_tid=18566], [object_number=93, tag_tid=11, entry_tid=18565]]], state=connected] 2024-08-31T22:22:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=92, tag_tid=11, entry_tid=18568' 2024-08-31T22:22:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=92, tag_tid=11, entry_tid=18568' 2024-08-31T22:22:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=92, tag_tid=11, entry_tid=18568' '!=' 'object_number=93, tag_tid=11, entry_tid=18453' 2024-08-31T22:22:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=92, tag_tid=11, entry_tid=18568' = 'object_number=93, tag_tid=11, entry_tid=18453' 2024-08-31T22:22:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=92, tag_tid=11, entry_tid=18568' 2024-08-31T22:22:06.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:06.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:06.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=92, tag_tid=11, entry_tid=18568' 2024-08-31T22:22:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=92, tag_tid=11, entry_tid=18568' 2024-08-31T22:22:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18568 2024-08-31T22:22:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 18568 2024-08-31T22:22:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=92, tag_tid=11, entry_tid=18680], [object_number=95, tag_tid=11, entry_tid=18679], [object_number=94, tag_tid=11, entry_tid=18678], [object_number=93, tag_tid=11, entry_tid=18677]]], state=connected] 2024-08-31T22:22:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=92, tag_tid=11, entry_tid=18680], [object_number=95, tag_tid=11, entry_tid=18679], [object_number=94, tag_tid=11, entry_tid=18678], [object_number=93, tag_tid=11, entry_tid=18677]]], state=connected] 2024-08-31T22:22:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=92, tag_tid=11, entry_tid=18680' 2024-08-31T22:22:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=92, tag_tid=11, entry_tid=18680' 2024-08-31T22:22:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=92, tag_tid=11, entry_tid=18680' '!=' 'object_number=92, tag_tid=11, entry_tid=18568' 2024-08-31T22:22:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=92, tag_tid=11, entry_tid=18680' = 'object_number=92, tag_tid=11, entry_tid=18568' 2024-08-31T22:22:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=92, tag_tid=11, entry_tid=18680' 2024-08-31T22:22:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=92, tag_tid=11, entry_tid=18680' 2024-08-31T22:22:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=92, tag_tid=11, entry_tid=18680' 2024-08-31T22:22:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18680 2024-08-31T22:22:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 18680 2024-08-31T22:22:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=94, tag_tid=11, entry_tid=18778], [object_number=93, tag_tid=11, entry_tid=18777], [object_number=92, tag_tid=11, entry_tid=18776], [object_number=95, tag_tid=11, entry_tid=18775]]], state=connected] 2024-08-31T22:22:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=94, tag_tid=11, entry_tid=18778], [object_number=93, tag_tid=11, entry_tid=18777], [object_number=92, tag_tid=11, entry_tid=18776], [object_number=95, tag_tid=11, entry_tid=18775]]], state=connected] 2024-08-31T22:22:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=94, tag_tid=11, entry_tid=18778' 2024-08-31T22:22:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=94, tag_tid=11, entry_tid=18778' 2024-08-31T22:22:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=94, tag_tid=11, entry_tid=18778' '!=' 'object_number=92, tag_tid=11, entry_tid=18680' 2024-08-31T22:22:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=94, tag_tid=11, entry_tid=18778' = 'object_number=92, tag_tid=11, entry_tid=18680' 2024-08-31T22:22:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=94, tag_tid=11, entry_tid=18778' 2024-08-31T22:22:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=94, tag_tid=11, entry_tid=18778' 2024-08-31T22:22:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=94, tag_tid=11, entry_tid=18778' 2024-08-31T22:22:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18778 2024-08-31T22:22:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 18778 2024-08-31T22:22:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:07.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:07.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=95, tag_tid=11, entry_tid=18883], [object_number=94, tag_tid=11, entry_tid=18882], [object_number=93, tag_tid=11, entry_tid=18881], [object_number=92, tag_tid=11, entry_tid=18880]]], state=connected] 2024-08-31T22:22:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=95, tag_tid=11, entry_tid=18883], [object_number=94, tag_tid=11, entry_tid=18882], [object_number=93, tag_tid=11, entry_tid=18881], [object_number=92, tag_tid=11, entry_tid=18880]]], state=connected] 2024-08-31T22:22:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:07.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=95, tag_tid=11, entry_tid=18883' 2024-08-31T22:22:07.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=95, tag_tid=11, entry_tid=18883' 2024-08-31T22:22:07.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=95, tag_tid=11, entry_tid=18883' '!=' 'object_number=94, tag_tid=11, entry_tid=18778' 2024-08-31T22:22:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=95, tag_tid=11, entry_tid=18883' = 'object_number=94, tag_tid=11, entry_tid=18778' 2024-08-31T22:22:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=95, tag_tid=11, entry_tid=18883' 2024-08-31T22:22:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=95, tag_tid=11, entry_tid=18883' 2024-08-31T22:22:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:07.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=95, tag_tid=11, entry_tid=18883' 2024-08-31T22:22:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18883 2024-08-31T22:22:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 18883 2024-08-31T22:22:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:08.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:08.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:08.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:08.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:08.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:08.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=18989], [object_number=92, tag_tid=11, entry_tid=18988], [object_number=95, tag_tid=11, entry_tid=18987], [object_number=94, tag_tid=11, entry_tid=18986]]], state=connected] 2024-08-31T22:22:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=18989], [object_number=92, tag_tid=11, entry_tid=18988], [object_number=95, tag_tid=11, entry_tid=18987], [object_number=94, tag_tid=11, entry_tid=18986]]], state=connected] 2024-08-31T22:22:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=93, tag_tid=11, entry_tid=18989' 2024-08-31T22:22:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=93, tag_tid=11, entry_tid=18989' 2024-08-31T22:22:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=18989' '!=' 'object_number=95, tag_tid=11, entry_tid=18883' 2024-08-31T22:22:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=18989' = 'object_number=95, tag_tid=11, entry_tid=18883' 2024-08-31T22:22:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=93, tag_tid=11, entry_tid=18989' 2024-08-31T22:22:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=18989' 2024-08-31T22:22:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=18989' 2024-08-31T22:22:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18989 2024-08-31T22:22:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 18989 2024-08-31T22:22:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:08.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:08.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=92, tag_tid=11, entry_tid=19096], [object_number=95, tag_tid=11, entry_tid=19095], [object_number=94, tag_tid=11, entry_tid=19094], [object_number=93, tag_tid=11, entry_tid=19093]]], state=connected] 2024-08-31T22:22:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:08.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=92, tag_tid=11, entry_tid=19096], [object_number=95, tag_tid=11, entry_tid=19095], [object_number=94, tag_tid=11, entry_tid=19094], [object_number=93, tag_tid=11, entry_tid=19093]]], state=connected] 2024-08-31T22:22:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=92, tag_tid=11, entry_tid=19096' 2024-08-31T22:22:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=92, tag_tid=11, entry_tid=19096' 2024-08-31T22:22:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=92, tag_tid=11, entry_tid=19096' '!=' 'object_number=93, tag_tid=11, entry_tid=18989' 2024-08-31T22:22:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=92, tag_tid=11, entry_tid=19096' = 'object_number=93, tag_tid=11, entry_tid=18989' 2024-08-31T22:22:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=92, tag_tid=11, entry_tid=19096' 2024-08-31T22:22:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=92, tag_tid=11, entry_tid=19096' 2024-08-31T22:22:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:08.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:08.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=92, tag_tid=11, entry_tid=19096' 2024-08-31T22:22:08.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:08.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19096 2024-08-31T22:22:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 19096 2024-08-31T22:22:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:08.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:08.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:08.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:08.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:08.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:08.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:08.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:08.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:08.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:08.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:08.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:09.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=92, tag_tid=11, entry_tid=19200], [object_number=95, tag_tid=11, entry_tid=19199], [object_number=94, tag_tid=11, entry_tid=19198], [object_number=93, tag_tid=11, entry_tid=19197]]], state=connected] 2024-08-31T22:22:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=92, tag_tid=11, entry_tid=19200], [object_number=95, tag_tid=11, entry_tid=19199], [object_number=94, tag_tid=11, entry_tid=19198], [object_number=93, tag_tid=11, entry_tid=19197]]], state=connected] 2024-08-31T22:22:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=92, tag_tid=11, entry_tid=19200' 2024-08-31T22:22:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=92, tag_tid=11, entry_tid=19200' 2024-08-31T22:22:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=92, tag_tid=11, entry_tid=19200' '!=' 'object_number=92, tag_tid=11, entry_tid=19096' 2024-08-31T22:22:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=92, tag_tid=11, entry_tid=19200' = 'object_number=92, tag_tid=11, entry_tid=19096' 2024-08-31T22:22:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=92, tag_tid=11, entry_tid=19200' 2024-08-31T22:22:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=92, tag_tid=11, entry_tid=19200' 2024-08-31T22:22:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=92, tag_tid=11, entry_tid=19200' 2024-08-31T22:22:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19200 2024-08-31T22:22:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 19200 2024-08-31T22:22:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:09.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:09.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:09.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:09.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:09.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=19301], [object_number=92, tag_tid=11, entry_tid=19300], [object_number=95, tag_tid=11, entry_tid=19299], [object_number=94, tag_tid=11, entry_tid=19298]]], state=connected] 2024-08-31T22:22:09.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=19301], [object_number=92, tag_tid=11, entry_tid=19300], [object_number=95, tag_tid=11, entry_tid=19299], [object_number=94, tag_tid=11, entry_tid=19298]]], state=connected] 2024-08-31T22:22:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=93, tag_tid=11, entry_tid=19301' 2024-08-31T22:22:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=93, tag_tid=11, entry_tid=19301' 2024-08-31T22:22:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=19301' '!=' 'object_number=92, tag_tid=11, entry_tid=19200' 2024-08-31T22:22:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=19301' = 'object_number=92, tag_tid=11, entry_tid=19200' 2024-08-31T22:22:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=93, tag_tid=11, entry_tid=19301' 2024-08-31T22:22:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=19301' 2024-08-31T22:22:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=19301' 2024-08-31T22:22:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19301 2024-08-31T22:22:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 19301 2024-08-31T22:22:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=94, tag_tid=11, entry_tid=19410], [object_number=93, tag_tid=11, entry_tid=19409], [object_number=92, tag_tid=11, entry_tid=19408], [object_number=95, tag_tid=11, entry_tid=19407]]], state=connected] 2024-08-31T22:22:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=94, tag_tid=11, entry_tid=19410], [object_number=93, tag_tid=11, entry_tid=19409], [object_number=92, tag_tid=11, entry_tid=19408], [object_number=95, tag_tid=11, entry_tid=19407]]], state=connected] 2024-08-31T22:22:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=94, tag_tid=11, entry_tid=19410' 2024-08-31T22:22:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=94, tag_tid=11, entry_tid=19410' 2024-08-31T22:22:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=94, tag_tid=11, entry_tid=19410' '!=' 'object_number=93, tag_tid=11, entry_tid=19301' 2024-08-31T22:22:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=94, tag_tid=11, entry_tid=19410' = 'object_number=93, tag_tid=11, entry_tid=19301' 2024-08-31T22:22:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=94, tag_tid=11, entry_tid=19410' 2024-08-31T22:22:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=94, tag_tid=11, entry_tid=19410' 2024-08-31T22:22:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=94, tag_tid=11, entry_tid=19410' 2024-08-31T22:22:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19410 2024-08-31T22:22:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 19410 2024-08-31T22:22:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=19513], [object_number=92, tag_tid=11, entry_tid=19512], [object_number=95, tag_tid=11, entry_tid=19511], [object_number=94, tag_tid=11, entry_tid=19510]]], state=connected] 2024-08-31T22:22:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 23 2024-08-31T22:22:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=93, tag_tid=11, entry_tid=19513], [object_number=92, tag_tid=11, entry_tid=19512], [object_number=95, tag_tid=11, entry_tid=19511], [object_number=94, tag_tid=11, entry_tid=19510]]], state=connected] 2024-08-31T22:22:10.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=93, tag_tid=11, entry_tid=19513' 2024-08-31T22:22:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=93, tag_tid=11, entry_tid=19513' 2024-08-31T22:22:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=19513' '!=' 'object_number=94, tag_tid=11, entry_tid=19410' 2024-08-31T22:22:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=93, tag_tid=11, entry_tid=19513' = 'object_number=94, tag_tid=11, entry_tid=19410' 2024-08-31T22:22:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=93, tag_tid=11, entry_tid=19513' 2024-08-31T22:22:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=19513' 2024-08-31T22:22:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:10.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=93, tag_tid=11, entry_tid=19513' 2024-08-31T22:22:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:10.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19513 2024-08-31T22:22:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 19513 2024-08-31T22:22:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:11.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:11.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:11.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:11.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:11.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:11.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 24 2024-08-31T22:22:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=98, tag_tid=11, entry_tid=19606], [object_number=97, tag_tid=11, entry_tid=19605], [object_number=96, tag_tid=11, entry_tid=19604], [object_number=99, tag_tid=11, entry_tid=19603]]], state=connected] 2024-08-31T22:22:11.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 24 2024-08-31T22:22:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=98, tag_tid=11, entry_tid=19606], [object_number=97, tag_tid=11, entry_tid=19605], [object_number=96, tag_tid=11, entry_tid=19604], [object_number=99, tag_tid=11, entry_tid=19603]]], state=connected] 2024-08-31T22:22:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=98, tag_tid=11, entry_tid=19606' 2024-08-31T22:22:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=98, tag_tid=11, entry_tid=19606' 2024-08-31T22:22:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=98, tag_tid=11, entry_tid=19606' '!=' 'object_number=93, tag_tid=11, entry_tid=19513' 2024-08-31T22:22:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=98, tag_tid=11, entry_tid=19606' = 'object_number=93, tag_tid=11, entry_tid=19513' 2024-08-31T22:22:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=98, tag_tid=11, entry_tid=19606' 2024-08-31T22:22:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=98, tag_tid=11, entry_tid=19606' 2024-08-31T22:22:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=98, tag_tid=11, entry_tid=19606' 2024-08-31T22:22:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19606 2024-08-31T22:22:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 19606 2024-08-31T22:22:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:11.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:11.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:11.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:11.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 24 2024-08-31T22:22:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=99, tag_tid=11, entry_tid=19707], [object_number=98, tag_tid=11, entry_tid=19706], [object_number=97, tag_tid=11, entry_tid=19705], [object_number=96, tag_tid=11, entry_tid=19704]]], state=connected] 2024-08-31T22:22:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:11.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 24 2024-08-31T22:22:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=99, tag_tid=11, entry_tid=19707], [object_number=98, tag_tid=11, entry_tid=19706], [object_number=97, tag_tid=11, entry_tid=19705], [object_number=96, tag_tid=11, entry_tid=19704]]], state=connected] 2024-08-31T22:22:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=99, tag_tid=11, entry_tid=19707' 2024-08-31T22:22:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=99, tag_tid=11, entry_tid=19707' 2024-08-31T22:22:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=99, tag_tid=11, entry_tid=19707' '!=' 'object_number=98, tag_tid=11, entry_tid=19606' 2024-08-31T22:22:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=99, tag_tid=11, entry_tid=19707' = 'object_number=98, tag_tid=11, entry_tid=19606' 2024-08-31T22:22:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=99, tag_tid=11, entry_tid=19707' 2024-08-31T22:22:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=99, tag_tid=11, entry_tid=19707' 2024-08-31T22:22:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=99, tag_tid=11, entry_tid=19707' 2024-08-31T22:22:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19707 2024-08-31T22:22:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 19707 2024-08-31T22:22:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:12.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:12.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:12.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:12.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 24 2024-08-31T22:22:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=98, tag_tid=11, entry_tid=19786], [object_number=97, tag_tid=11, entry_tid=19785], [object_number=96, tag_tid=11, entry_tid=19784], [object_number=99, tag_tid=11, entry_tid=19783]]], state=connected] 2024-08-31T22:22:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:12.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:12.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:12.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:12.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:12.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:12.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:12.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 24 2024-08-31T22:22:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=98, tag_tid=11, entry_tid=19786], [object_number=97, tag_tid=11, entry_tid=19785], [object_number=96, tag_tid=11, entry_tid=19784], [object_number=99, tag_tid=11, entry_tid=19783]]], state=connected] 2024-08-31T22:22:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:12.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=98, tag_tid=11, entry_tid=19786' 2024-08-31T22:22:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=98, tag_tid=11, entry_tid=19786' 2024-08-31T22:22:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=98, tag_tid=11, entry_tid=19786' '!=' 'object_number=99, tag_tid=11, entry_tid=19707' 2024-08-31T22:22:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=98, tag_tid=11, entry_tid=19786' = 'object_number=99, tag_tid=11, entry_tid=19707' 2024-08-31T22:22:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=98, tag_tid=11, entry_tid=19786' 2024-08-31T22:22:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=98, tag_tid=11, entry_tid=19786' 2024-08-31T22:22:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=98, tag_tid=11, entry_tid=19786' 2024-08-31T22:22:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19786 2024-08-31T22:22:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 19786 2024-08-31T22:22:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 24 2024-08-31T22:22:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=98, tag_tid=11, entry_tid=19798], [object_number=97, tag_tid=11, entry_tid=19797], [object_number=96, tag_tid=11, entry_tid=19796], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 24 2024-08-31T22:22:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=98, tag_tid=11, entry_tid=19798], [object_number=97, tag_tid=11, entry_tid=19797], [object_number=96, tag_tid=11, entry_tid=19796], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=98, tag_tid=11, entry_tid=19798' 2024-08-31T22:22:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=98, tag_tid=11, entry_tid=19798' 2024-08-31T22:22:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=98, tag_tid=11, entry_tid=19798' '!=' 'object_number=98, tag_tid=11, entry_tid=19786' 2024-08-31T22:22:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:22:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=98, tag_tid=11, entry_tid=19798' = 'object_number=98, tag_tid=11, entry_tid=19786' 2024-08-31T22:22:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=98, tag_tid=11, entry_tid=19798' 2024-08-31T22:22:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=12 2024-08-31T22:22:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=98, tag_tid=11, entry_tid=19798' 2024-08-31T22:22:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:22:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=11 2024-08-31T22:22:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:22:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=98, tag_tid=11, entry_tid=19798' 2024-08-31T22:22:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:22:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:22:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19798 2024-08-31T22:22:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 12 = 11 -a 1 -le 19798 2024-08-31T22:22:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:22:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:22:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:13.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:13.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 24 2024-08-31T22:22:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=98, tag_tid=11, entry_tid=19798], [object_number=97, tag_tid=11, entry_tid=19797], [object_number=96, tag_tid=11, entry_tid=19796], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 24 2024-08-31T22:22:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=98, tag_tid=11, entry_tid=19798], [object_number=97, tag_tid=11, entry_tid=19797], [object_number=96, tag_tid=11, entry_tid=19796], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:13.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=98, tag_tid=11, entry_tid=19798' 2024-08-31T22:22:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=98, tag_tid=11, entry_tid=19798' 2024-08-31T22:22:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=98, tag_tid=11, entry_tid=19798' '!=' 'object_number=98, tag_tid=11, entry_tid=19798' 2024-08-31T22:22:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:22:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.4 2024-08-31T22:22:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:22:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:22:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:22:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:22:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:22:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:22:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:22:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:22:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:22:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:22:13.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:22:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:22:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:22:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:22:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:22:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:22:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:22:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:22:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:22:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:22:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:22:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:22:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:22:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:22:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:22:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:22:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:22:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:22:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:22:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:22:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:22:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:22:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:22:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:22:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:22:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:22:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:22:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:22:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:22:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:22:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:22:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:22:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:22:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:22:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:22:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:13.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 24 2024-08-31T22:22:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:13.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:22:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:22:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:22:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:22:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:22:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:22:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:22:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:22:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:22:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:22:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 24 2024-08-31T22:22:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 24 2024-08-31T22:22:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:22:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=97, tag_tid=12, entry_tid=1], [object_number=96, tag_tid=12, entry_tid=0], [object_number=98, tag_tid=11, entry_tid=19798], [object_number=99, tag_tid=11, entry_tid=19795]]], state=connected] 2024-08-31T22:22:13.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:22:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=97, tag_tid=12, entry_tid=1' -a 'object_number=97, tag_tid=12, entry_tid=1' = 'object_number=97, tag_tid=12, entry_tid=1' 2024-08-31T22:22:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-08-31T22:22:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap4 2024-08-31T22:22:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:22:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:22:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:22:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-08-31T22:22:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'test': 2024-08-31T22:22:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:22:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:22:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 7 2024-08-31T22:22:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a87b4c6a1b 2024-08-31T22:22:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a87b4c6a1b 2024-08-31T22:22:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:22:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:22:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:22:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:22:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:14:57 2024 2024-08-31T22:22:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:20:18 2024 2024-08-31T22:22:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:22:04 2024 2024-08-31T22:22:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:22:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a87b4c6a1b 2024-08-31T22:22:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:22:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:22:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:22:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:22:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n snap3-clean ']' 2024-08-31T22:22:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap3-clean 2024-08-31T22:22:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap3-clean 2024-08-31T22:22:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:22:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:22:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:22:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:22:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:22:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:22:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:22:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:22:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:22:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:22:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap4 2024-08-31T22:22:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap4 2024-08-31T22:22:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:22:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:22:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:22:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:22:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:22:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:22:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:22:19.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:22:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:22:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:22:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ clean_snap_name=snap4-clean 2024-08-31T22:22:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-08-31T22:22:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:22:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap4-clean 2024-08-31T22:22:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-08-31T22:22:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:22:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror test 2024-08-31T22:22:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:22:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:22:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=14 2024-08-31T22:22:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:22:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-08-31T22:22:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 23750656 bytes 2024-08-31T22:22:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 64893952 bytes 2024-08-31T22:22:24.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 104502272 bytes 2024-08-31T22:22:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 158268416 bytes 2024-08-31T22:22:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:500 IOs, 199249920 bytes 2024-08-31T22:22:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:600 IOs, 272113664 bytes 2024-08-31T22:22:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:700 IOs, 280384512 bytes 2024-08-31T22:22:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:800 IOs, 317053952 bytes 2024-08-31T22:22:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:900 IOs, 352014336 bytes 2024-08-31T22:22:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:22:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:22:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:22:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-08-31T22:22:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:22:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:22:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:22:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:22:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:22:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-08-31T22:22:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:22:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:22:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:22:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:22:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:22:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-08-31T22:22:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='test: 2024-08-31T22:22:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: global_id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:22:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+replaying 2024-08-31T22:22:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: replaying, {"bytes_per_second":2198846.37,"entries_behind_primary":1,"entries_per_second":138.53,"non_primary_position":{"entry_tid":1,"object_number":97,"tag_tid":12},"primary_position":{"entry_tid":1,"object_number":97,"tag_tid":13},"seconds_until_synced":0} 2024-08-31T22:22:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: service: mirror.0 on smithi032 2024-08-31T22:22:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:22:25 2024-08-31T22:22:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: peer_sites: 2024-08-31T22:22:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: name: cluster2 2024-08-31T22:22:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+stopped 2024-08-31T22:22:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: local image is primary 2024-08-31T22:22:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:22:30' 2024-08-31T22:22:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ state: .*up+replaying' 2024-08-31T22:22:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: state: up+replaying 2024-08-31T22:22:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ description: .*' 2024-08-31T22:22:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: description: replaying, {"bytes_per_second":2198846.37,"entries_behind_primary":1,"entries_per_second":138.53,"non_primary_position":{"entry_tid":1,"object_number":97,"tag_tid":12},"primary_position":{"entry_tid":1,"object_number":97,"tag_tid":13},"seconds_until_synced":0} 2024-08-31T22:22:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:22:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:22:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^up+' 2024-08-31T22:22:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:22:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep 'service: *mirror..* on ' 2024-08-31T22:22:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: service: mirror.0 on smithi032 2024-08-31T22:22:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update 2024-08-31T22:22:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-08-31T22:22:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:22:25' 2024-08-31T22:22:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-08-31 22:22:25' 2024-08-31T22:22:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:22:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:22:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'prev_last_update=2024-08-31 22:22:25' 2024-08-31T22:22:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:22:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:22:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-08-31T22:22:35.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='OKOKOK142574264mirror.0smithi03219.1.1-244-g3091dfc0trueOK42614267mirror.1smithi03219.1.1-244-g3091dfc0falseOKtestdd2c705e-25cf-4565-981f-d48de48eee84up+replayingreplaying, {"bytes_per_second":2198846.37,"entries_behind_primary":1,"entries_per_second":138.53,"non_primary_position":{"entry_tid":1,"object_number":97,"tag_tid":12},"primary_position":{"entry_tid":1,"object_number":97,"tag_tid":13},"seconds_until_synced":0}42574264mirror.0smithi0322024-08-31 22:22:25cluster24ca32b00-2164-486e-837e-e93795a16af1up+stoppedlocal image is primary2024-08-31 22:22:35' 2024-08-31T22:22:35.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update state 2024-08-31T22:22:35.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-08-31T22:22:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:22:25' 2024-08-31T22:22:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-08-31T22:22:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ state=up+replaying 2024-08-31T22:22:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:22:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep up+replaying 2024-08-31T22:22:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:22:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:22:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap5 2024-08-31T22:22:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap5 2024-08-31T22:22:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:22:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:22:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:22:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap5 2024-08-31T22:22:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:23:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-08-31T22:23:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-08-31T22:23:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:23:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:23:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:23:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror test started 2024-08-31T22:23:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:23:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:23:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:23:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-08-31T22:23:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:23:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:23:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:23:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:23:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:23:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:23:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:23:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-08-31T22:23:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:23:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:23:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:23:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:23:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:23:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:23:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:23:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:23:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:23:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:23:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:23:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:23:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:23:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:23:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:23:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:23:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-08-31T22:23:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:23:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:23:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:23:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:23:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:23:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:23:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:23:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:23:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:23:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:23:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:23:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:23:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:23:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:23:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:23:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:23:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=7569], [object_number=104, tag_tid=14, entry_tid=7568], [object_number=107, tag_tid=14, entry_tid=7567], [object_number=106, tag_tid=14, entry_tid=7566]]], state=connected] 2024-08-31T22:23:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:09.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:09.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=7569], [object_number=104, tag_tid=14, entry_tid=7568], [object_number=107, tag_tid=14, entry_tid=7567], [object_number=106, tag_tid=14, entry_tid=7566]]], state=connected] 2024-08-31T22:23:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=105, tag_tid=14, entry_tid=7569' 2024-08-31T22:23:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=105, tag_tid=14, entry_tid=7569' 2024-08-31T22:23:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=7569' '!=' '' 2024-08-31T22:23:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=7569' = '' 2024-08-31T22:23:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=105, tag_tid=14, entry_tid=7569' 2024-08-31T22:23:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=7569' 2024-08-31T22:23:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=7569' 2024-08-31T22:23:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7569 2024-08-31T22:23:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 7569 2024-08-31T22:23:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=7693], [object_number=104, tag_tid=14, entry_tid=7692], [object_number=107, tag_tid=14, entry_tid=7691], [object_number=106, tag_tid=14, entry_tid=7690]]], state=connected] 2024-08-31T22:23:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=7693], [object_number=104, tag_tid=14, entry_tid=7692], [object_number=107, tag_tid=14, entry_tid=7691], [object_number=106, tag_tid=14, entry_tid=7690]]], state=connected] 2024-08-31T22:23:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=105, tag_tid=14, entry_tid=7693' 2024-08-31T22:23:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=105, tag_tid=14, entry_tid=7693' 2024-08-31T22:23:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=7693' '!=' 'object_number=105, tag_tid=14, entry_tid=7569' 2024-08-31T22:23:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=7693' = 'object_number=105, tag_tid=14, entry_tid=7569' 2024-08-31T22:23:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=105, tag_tid=14, entry_tid=7693' 2024-08-31T22:23:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=7693' 2024-08-31T22:23:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=7693' 2024-08-31T22:23:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7693 2024-08-31T22:23:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 7693 2024-08-31T22:23:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=7785], [object_number=104, tag_tid=14, entry_tid=7784], [object_number=107, tag_tid=14, entry_tid=7783], [object_number=106, tag_tid=14, entry_tid=7782]]], state=connected] 2024-08-31T22:23:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=7785], [object_number=104, tag_tid=14, entry_tid=7784], [object_number=107, tag_tid=14, entry_tid=7783], [object_number=106, tag_tid=14, entry_tid=7782]]], state=connected] 2024-08-31T22:23:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=105, tag_tid=14, entry_tid=7785' 2024-08-31T22:23:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=105, tag_tid=14, entry_tid=7785' 2024-08-31T22:23:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=7785' '!=' 'object_number=105, tag_tid=14, entry_tid=7693' 2024-08-31T22:23:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=7785' = 'object_number=105, tag_tid=14, entry_tid=7693' 2024-08-31T22:23:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=105, tag_tid=14, entry_tid=7785' 2024-08-31T22:23:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=7785' 2024-08-31T22:23:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:10.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:10.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:10.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=7785' 2024-08-31T22:23:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7785 2024-08-31T22:23:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 7785 2024-08-31T22:23:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=106, tag_tid=14, entry_tid=7882], [object_number=105, tag_tid=14, entry_tid=7881], [object_number=104, tag_tid=14, entry_tid=7880], [object_number=107, tag_tid=14, entry_tid=7879]]], state=connected] 2024-08-31T22:23:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:11.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:11.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:11.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:11.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=106, tag_tid=14, entry_tid=7882], [object_number=105, tag_tid=14, entry_tid=7881], [object_number=104, tag_tid=14, entry_tid=7880], [object_number=107, tag_tid=14, entry_tid=7879]]], state=connected] 2024-08-31T22:23:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:11.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=106, tag_tid=14, entry_tid=7882' 2024-08-31T22:23:11.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=106, tag_tid=14, entry_tid=7882' 2024-08-31T22:23:11.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=106, tag_tid=14, entry_tid=7882' '!=' 'object_number=105, tag_tid=14, entry_tid=7785' 2024-08-31T22:23:11.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:11.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=106, tag_tid=14, entry_tid=7882' = 'object_number=105, tag_tid=14, entry_tid=7785' 2024-08-31T22:23:11.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=106, tag_tid=14, entry_tid=7882' 2024-08-31T22:23:11.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:11.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:11.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:11.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=106, tag_tid=14, entry_tid=7882' 2024-08-31T22:23:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=106, tag_tid=14, entry_tid=7882' 2024-08-31T22:23:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7882 2024-08-31T22:23:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 7882 2024-08-31T22:23:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:11.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:11.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=7965], [object_number=104, tag_tid=14, entry_tid=7964], [object_number=107, tag_tid=14, entry_tid=7963], [object_number=106, tag_tid=14, entry_tid=7962]]], state=connected] 2024-08-31T22:23:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:11.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:11.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:11.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:11.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:11.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=7965], [object_number=104, tag_tid=14, entry_tid=7964], [object_number=107, tag_tid=14, entry_tid=7963], [object_number=106, tag_tid=14, entry_tid=7962]]], state=connected] 2024-08-31T22:23:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=105, tag_tid=14, entry_tid=7965' 2024-08-31T22:23:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=105, tag_tid=14, entry_tid=7965' 2024-08-31T22:23:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=7965' '!=' 'object_number=106, tag_tid=14, entry_tid=7882' 2024-08-31T22:23:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=7965' = 'object_number=106, tag_tid=14, entry_tid=7882' 2024-08-31T22:23:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=105, tag_tid=14, entry_tid=7965' 2024-08-31T22:23:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=7965' 2024-08-31T22:23:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=7965' 2024-08-31T22:23:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7965 2024-08-31T22:23:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 7965 2024-08-31T22:23:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=106, tag_tid=14, entry_tid=8050], [object_number=105, tag_tid=14, entry_tid=8049], [object_number=104, tag_tid=14, entry_tid=8048], [object_number=107, tag_tid=14, entry_tid=8047]]], state=connected] 2024-08-31T22:23:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:12.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:12.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:12.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:12.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:12.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:12.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=106, tag_tid=14, entry_tid=8050], [object_number=105, tag_tid=14, entry_tid=8049], [object_number=104, tag_tid=14, entry_tid=8048], [object_number=107, tag_tid=14, entry_tid=8047]]], state=connected] 2024-08-31T22:23:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=106, tag_tid=14, entry_tid=8050' 2024-08-31T22:23:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=106, tag_tid=14, entry_tid=8050' 2024-08-31T22:23:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=106, tag_tid=14, entry_tid=8050' '!=' 'object_number=105, tag_tid=14, entry_tid=7965' 2024-08-31T22:23:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=106, tag_tid=14, entry_tid=8050' = 'object_number=105, tag_tid=14, entry_tid=7965' 2024-08-31T22:23:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=106, tag_tid=14, entry_tid=8050' 2024-08-31T22:23:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=106, tag_tid=14, entry_tid=8050' 2024-08-31T22:23:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=106, tag_tid=14, entry_tid=8050' 2024-08-31T22:23:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8050 2024-08-31T22:23:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 8050 2024-08-31T22:23:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:12.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:12.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:12.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=8137], [object_number=104, tag_tid=14, entry_tid=8136], [object_number=107, tag_tid=14, entry_tid=8135], [object_number=106, tag_tid=14, entry_tid=8134]]], state=connected] 2024-08-31T22:23:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=8137], [object_number=104, tag_tid=14, entry_tid=8136], [object_number=107, tag_tid=14, entry_tid=8135], [object_number=106, tag_tid=14, entry_tid=8134]]], state=connected] 2024-08-31T22:23:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=105, tag_tid=14, entry_tid=8137' 2024-08-31T22:23:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=105, tag_tid=14, entry_tid=8137' 2024-08-31T22:23:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=8137' '!=' 'object_number=106, tag_tid=14, entry_tid=8050' 2024-08-31T22:23:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=8137' = 'object_number=106, tag_tid=14, entry_tid=8050' 2024-08-31T22:23:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=105, tag_tid=14, entry_tid=8137' 2024-08-31T22:23:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=8137' 2024-08-31T22:23:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=8137' 2024-08-31T22:23:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8137 2024-08-31T22:23:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 8137 2024-08-31T22:23:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=8233], [object_number=104, tag_tid=14, entry_tid=8232], [object_number=107, tag_tid=14, entry_tid=8231], [object_number=106, tag_tid=14, entry_tid=8230]]], state=connected] 2024-08-31T22:23:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:13.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:13.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:13.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=8233], [object_number=104, tag_tid=14, entry_tid=8232], [object_number=107, tag_tid=14, entry_tid=8231], [object_number=106, tag_tid=14, entry_tid=8230]]], state=connected] 2024-08-31T22:23:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=105, tag_tid=14, entry_tid=8233' 2024-08-31T22:23:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=105, tag_tid=14, entry_tid=8233' 2024-08-31T22:23:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=8233' '!=' 'object_number=105, tag_tid=14, entry_tid=8137' 2024-08-31T22:23:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=8233' = 'object_number=105, tag_tid=14, entry_tid=8137' 2024-08-31T22:23:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=105, tag_tid=14, entry_tid=8233' 2024-08-31T22:23:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=8233' 2024-08-31T22:23:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=8233' 2024-08-31T22:23:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8233 2024-08-31T22:23:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 8233 2024-08-31T22:23:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=104, tag_tid=14, entry_tid=8324], [object_number=107, tag_tid=14, entry_tid=8323], [object_number=106, tag_tid=14, entry_tid=8322], [object_number=105, tag_tid=14, entry_tid=8321]]], state=connected] 2024-08-31T22:23:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=104, tag_tid=14, entry_tid=8324], [object_number=107, tag_tid=14, entry_tid=8323], [object_number=106, tag_tid=14, entry_tid=8322], [object_number=105, tag_tid=14, entry_tid=8321]]], state=connected] 2024-08-31T22:23:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=104, tag_tid=14, entry_tid=8324' 2024-08-31T22:23:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=104, tag_tid=14, entry_tid=8324' 2024-08-31T22:23:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=104, tag_tid=14, entry_tid=8324' '!=' 'object_number=105, tag_tid=14, entry_tid=8233' 2024-08-31T22:23:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=104, tag_tid=14, entry_tid=8324' = 'object_number=105, tag_tid=14, entry_tid=8233' 2024-08-31T22:23:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=104, tag_tid=14, entry_tid=8324' 2024-08-31T22:23:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=104, tag_tid=14, entry_tid=8324' 2024-08-31T22:23:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=104, tag_tid=14, entry_tid=8324' 2024-08-31T22:23:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8324 2024-08-31T22:23:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 8324 2024-08-31T22:23:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=106, tag_tid=14, entry_tid=8418], [object_number=105, tag_tid=14, entry_tid=8417], [object_number=104, tag_tid=14, entry_tid=8416], [object_number=107, tag_tid=14, entry_tid=8415]]], state=connected] 2024-08-31T22:23:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=106, tag_tid=14, entry_tid=8418], [object_number=105, tag_tid=14, entry_tid=8417], [object_number=104, tag_tid=14, entry_tid=8416], [object_number=107, tag_tid=14, entry_tid=8415]]], state=connected] 2024-08-31T22:23:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=106, tag_tid=14, entry_tid=8418' 2024-08-31T22:23:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=106, tag_tid=14, entry_tid=8418' 2024-08-31T22:23:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=106, tag_tid=14, entry_tid=8418' '!=' 'object_number=104, tag_tid=14, entry_tid=8324' 2024-08-31T22:23:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=106, tag_tid=14, entry_tid=8418' = 'object_number=104, tag_tid=14, entry_tid=8324' 2024-08-31T22:23:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=106, tag_tid=14, entry_tid=8418' 2024-08-31T22:23:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=106, tag_tid=14, entry_tid=8418' 2024-08-31T22:23:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=106, tag_tid=14, entry_tid=8418' 2024-08-31T22:23:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8418 2024-08-31T22:23:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 8418 2024-08-31T22:23:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:14.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:14.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:14.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=104, tag_tid=14, entry_tid=8508], [object_number=107, tag_tid=14, entry_tid=8507], [object_number=106, tag_tid=14, entry_tid=8506], [object_number=105, tag_tid=14, entry_tid=8505]]], state=connected] 2024-08-31T22:23:14.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=104, tag_tid=14, entry_tid=8508], [object_number=107, tag_tid=14, entry_tid=8507], [object_number=106, tag_tid=14, entry_tid=8506], [object_number=105, tag_tid=14, entry_tid=8505]]], state=connected] 2024-08-31T22:23:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=104, tag_tid=14, entry_tid=8508' 2024-08-31T22:23:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=104, tag_tid=14, entry_tid=8508' 2024-08-31T22:23:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=104, tag_tid=14, entry_tid=8508' '!=' 'object_number=106, tag_tid=14, entry_tid=8418' 2024-08-31T22:23:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=104, tag_tid=14, entry_tid=8508' = 'object_number=106, tag_tid=14, entry_tid=8418' 2024-08-31T22:23:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=104, tag_tid=14, entry_tid=8508' 2024-08-31T22:23:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:14.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=104, tag_tid=14, entry_tid=8508' 2024-08-31T22:23:14.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=104, tag_tid=14, entry_tid=8508' 2024-08-31T22:23:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8508 2024-08-31T22:23:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 8508 2024-08-31T22:23:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:14.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:14.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:14.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:14.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:14.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:14.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:14.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:14.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:14.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:14.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:14.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=8601], [object_number=104, tag_tid=14, entry_tid=8600], [object_number=107, tag_tid=14, entry_tid=8599], [object_number=106, tag_tid=14, entry_tid=8598]]], state=connected] 2024-08-31T22:23:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=8601], [object_number=104, tag_tid=14, entry_tid=8600], [object_number=107, tag_tid=14, entry_tid=8599], [object_number=106, tag_tid=14, entry_tid=8598]]], state=connected] 2024-08-31T22:23:15.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=105, tag_tid=14, entry_tid=8601' 2024-08-31T22:23:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=105, tag_tid=14, entry_tid=8601' 2024-08-31T22:23:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=8601' '!=' 'object_number=104, tag_tid=14, entry_tid=8508' 2024-08-31T22:23:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=8601' = 'object_number=104, tag_tid=14, entry_tid=8508' 2024-08-31T22:23:15.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=105, tag_tid=14, entry_tid=8601' 2024-08-31T22:23:15.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:15.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:15.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:15.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=8601' 2024-08-31T22:23:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:15.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:15.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:15.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:15.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=8601' 2024-08-31T22:23:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:15.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8601 2024-08-31T22:23:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 8601 2024-08-31T22:23:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:15.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:15.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:15.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=8737], [object_number=104, tag_tid=14, entry_tid=8736], [object_number=107, tag_tid=14, entry_tid=8735], [object_number=106, tag_tid=14, entry_tid=8734]]], state=connected] 2024-08-31T22:23:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=8737], [object_number=104, tag_tid=14, entry_tid=8736], [object_number=107, tag_tid=14, entry_tid=8735], [object_number=106, tag_tid=14, entry_tid=8734]]], state=connected] 2024-08-31T22:23:15.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=105, tag_tid=14, entry_tid=8737' 2024-08-31T22:23:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=105, tag_tid=14, entry_tid=8737' 2024-08-31T22:23:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=8737' '!=' 'object_number=105, tag_tid=14, entry_tid=8601' 2024-08-31T22:23:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=8737' = 'object_number=105, tag_tid=14, entry_tid=8601' 2024-08-31T22:23:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=105, tag_tid=14, entry_tid=8737' 2024-08-31T22:23:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=8737' 2024-08-31T22:23:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=8737' 2024-08-31T22:23:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8737 2024-08-31T22:23:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 8737 2024-08-31T22:23:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:16.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:16.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:16.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:16.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:16.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:16.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:16.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:16.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:16.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:16.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:16.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:16.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=107, tag_tid=14, entry_tid=8823], [object_number=106, tag_tid=14, entry_tid=8822], [object_number=105, tag_tid=14, entry_tid=8821], [object_number=104, tag_tid=14, entry_tid=8820]]], state=connected] 2024-08-31T22:23:16.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:16.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=107, tag_tid=14, entry_tid=8823], [object_number=106, tag_tid=14, entry_tid=8822], [object_number=105, tag_tid=14, entry_tid=8821], [object_number=104, tag_tid=14, entry_tid=8820]]], state=connected] 2024-08-31T22:23:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=107, tag_tid=14, entry_tid=8823' 2024-08-31T22:23:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=107, tag_tid=14, entry_tid=8823' 2024-08-31T22:23:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=107, tag_tid=14, entry_tid=8823' '!=' 'object_number=105, tag_tid=14, entry_tid=8737' 2024-08-31T22:23:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=107, tag_tid=14, entry_tid=8823' = 'object_number=105, tag_tid=14, entry_tid=8737' 2024-08-31T22:23:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=107, tag_tid=14, entry_tid=8823' 2024-08-31T22:23:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=107, tag_tid=14, entry_tid=8823' 2024-08-31T22:23:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:16.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:16.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=107, tag_tid=14, entry_tid=8823' 2024-08-31T22:23:16.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:16.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8823 2024-08-31T22:23:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 8823 2024-08-31T22:23:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=8909], [object_number=104, tag_tid=14, entry_tid=8908], [object_number=107, tag_tid=14, entry_tid=8907], [object_number=106, tag_tid=14, entry_tid=8906]]], state=connected] 2024-08-31T22:23:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=8909], [object_number=104, tag_tid=14, entry_tid=8908], [object_number=107, tag_tid=14, entry_tid=8907], [object_number=106, tag_tid=14, entry_tid=8906]]], state=connected] 2024-08-31T22:23:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=105, tag_tid=14, entry_tid=8909' 2024-08-31T22:23:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=105, tag_tid=14, entry_tid=8909' 2024-08-31T22:23:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=8909' '!=' 'object_number=107, tag_tid=14, entry_tid=8823' 2024-08-31T22:23:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=8909' = 'object_number=107, tag_tid=14, entry_tid=8823' 2024-08-31T22:23:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=105, tag_tid=14, entry_tid=8909' 2024-08-31T22:23:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=8909' 2024-08-31T22:23:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=8909' 2024-08-31T22:23:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:16.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8909 2024-08-31T22:23:16.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 8909 2024-08-31T22:23:16.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:16.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:16.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:17.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:17.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:17.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:17.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=107, tag_tid=14, entry_tid=8995], [object_number=106, tag_tid=14, entry_tid=8994], [object_number=105, tag_tid=14, entry_tid=8993], [object_number=104, tag_tid=14, entry_tid=8992]]], state=connected] 2024-08-31T22:23:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=107, tag_tid=14, entry_tid=8995], [object_number=106, tag_tid=14, entry_tid=8994], [object_number=105, tag_tid=14, entry_tid=8993], [object_number=104, tag_tid=14, entry_tid=8992]]], state=connected] 2024-08-31T22:23:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=107, tag_tid=14, entry_tid=8995' 2024-08-31T22:23:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=107, tag_tid=14, entry_tid=8995' 2024-08-31T22:23:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=107, tag_tid=14, entry_tid=8995' '!=' 'object_number=105, tag_tid=14, entry_tid=8909' 2024-08-31T22:23:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=107, tag_tid=14, entry_tid=8995' = 'object_number=105, tag_tid=14, entry_tid=8909' 2024-08-31T22:23:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=107, tag_tid=14, entry_tid=8995' 2024-08-31T22:23:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=107, tag_tid=14, entry_tid=8995' 2024-08-31T22:23:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=107, tag_tid=14, entry_tid=8995' 2024-08-31T22:23:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8995 2024-08-31T22:23:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 8995 2024-08-31T22:23:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=104, tag_tid=14, entry_tid=9104], [object_number=107, tag_tid=14, entry_tid=9103], [object_number=106, tag_tid=14, entry_tid=9102], [object_number=105, tag_tid=14, entry_tid=9101]]], state=connected] 2024-08-31T22:23:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:17.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=104, tag_tid=14, entry_tid=9104], [object_number=107, tag_tid=14, entry_tid=9103], [object_number=106, tag_tid=14, entry_tid=9102], [object_number=105, tag_tid=14, entry_tid=9101]]], state=connected] 2024-08-31T22:23:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=104, tag_tid=14, entry_tid=9104' 2024-08-31T22:23:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=104, tag_tid=14, entry_tid=9104' 2024-08-31T22:23:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=104, tag_tid=14, entry_tid=9104' '!=' 'object_number=107, tag_tid=14, entry_tid=8995' 2024-08-31T22:23:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=104, tag_tid=14, entry_tid=9104' = 'object_number=107, tag_tid=14, entry_tid=8995' 2024-08-31T22:23:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=104, tag_tid=14, entry_tid=9104' 2024-08-31T22:23:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=104, tag_tid=14, entry_tid=9104' 2024-08-31T22:23:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=104, tag_tid=14, entry_tid=9104' 2024-08-31T22:23:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9104 2024-08-31T22:23:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 9104 2024-08-31T22:23:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:18.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:18.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:18.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=106, tag_tid=14, entry_tid=9202], [object_number=105, tag_tid=14, entry_tid=9201], [object_number=104, tag_tid=14, entry_tid=9200], [object_number=107, tag_tid=14, entry_tid=9199]]], state=connected] 2024-08-31T22:23:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=106, tag_tid=14, entry_tid=9202], [object_number=105, tag_tid=14, entry_tid=9201], [object_number=104, tag_tid=14, entry_tid=9200], [object_number=107, tag_tid=14, entry_tid=9199]]], state=connected] 2024-08-31T22:23:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=106, tag_tid=14, entry_tid=9202' 2024-08-31T22:23:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=106, tag_tid=14, entry_tid=9202' 2024-08-31T22:23:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=106, tag_tid=14, entry_tid=9202' '!=' 'object_number=104, tag_tid=14, entry_tid=9104' 2024-08-31T22:23:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=106, tag_tid=14, entry_tid=9202' = 'object_number=104, tag_tid=14, entry_tid=9104' 2024-08-31T22:23:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=106, tag_tid=14, entry_tid=9202' 2024-08-31T22:23:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=106, tag_tid=14, entry_tid=9202' 2024-08-31T22:23:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=106, tag_tid=14, entry_tid=9202' 2024-08-31T22:23:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9202 2024-08-31T22:23:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 9202 2024-08-31T22:23:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:18.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:18.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:18.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=9293], [object_number=104, tag_tid=14, entry_tid=9292], [object_number=107, tag_tid=14, entry_tid=9291], [object_number=106, tag_tid=14, entry_tid=9290]]], state=connected] 2024-08-31T22:23:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=9293], [object_number=104, tag_tid=14, entry_tid=9292], [object_number=107, tag_tid=14, entry_tid=9291], [object_number=106, tag_tid=14, entry_tid=9290]]], state=connected] 2024-08-31T22:23:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=105, tag_tid=14, entry_tid=9293' 2024-08-31T22:23:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=105, tag_tid=14, entry_tid=9293' 2024-08-31T22:23:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=9293' '!=' 'object_number=106, tag_tid=14, entry_tid=9202' 2024-08-31T22:23:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=9293' = 'object_number=106, tag_tid=14, entry_tid=9202' 2024-08-31T22:23:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=105, tag_tid=14, entry_tid=9293' 2024-08-31T22:23:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=9293' 2024-08-31T22:23:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:18.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=9293' 2024-08-31T22:23:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9293 2024-08-31T22:23:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 9293 2024-08-31T22:23:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=9385], [object_number=104, tag_tid=14, entry_tid=9384], [object_number=107, tag_tid=14, entry_tid=9383], [object_number=106, tag_tid=14, entry_tid=9382]]], state=connected] 2024-08-31T22:23:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:19.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:19.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:19.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:19.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=9385], [object_number=104, tag_tid=14, entry_tid=9384], [object_number=107, tag_tid=14, entry_tid=9383], [object_number=106, tag_tid=14, entry_tid=9382]]], state=connected] 2024-08-31T22:23:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=105, tag_tid=14, entry_tid=9385' 2024-08-31T22:23:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=105, tag_tid=14, entry_tid=9385' 2024-08-31T22:23:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=9385' '!=' 'object_number=105, tag_tid=14, entry_tid=9293' 2024-08-31T22:23:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=9385' = 'object_number=105, tag_tid=14, entry_tid=9293' 2024-08-31T22:23:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=105, tag_tid=14, entry_tid=9385' 2024-08-31T22:23:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=9385' 2024-08-31T22:23:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:19.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=9385' 2024-08-31T22:23:19.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:19.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9385 2024-08-31T22:23:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 9385 2024-08-31T22:23:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=106, tag_tid=14, entry_tid=9478], [object_number=105, tag_tid=14, entry_tid=9477], [object_number=104, tag_tid=14, entry_tid=9476], [object_number=107, tag_tid=14, entry_tid=9475]]], state=connected] 2024-08-31T22:23:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=106, tag_tid=14, entry_tid=9478], [object_number=105, tag_tid=14, entry_tid=9477], [object_number=104, tag_tid=14, entry_tid=9476], [object_number=107, tag_tid=14, entry_tid=9475]]], state=connected] 2024-08-31T22:23:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=106, tag_tid=14, entry_tid=9478' 2024-08-31T22:23:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=106, tag_tid=14, entry_tid=9478' 2024-08-31T22:23:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=106, tag_tid=14, entry_tid=9478' '!=' 'object_number=105, tag_tid=14, entry_tid=9385' 2024-08-31T22:23:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=106, tag_tid=14, entry_tid=9478' = 'object_number=105, tag_tid=14, entry_tid=9385' 2024-08-31T22:23:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=106, tag_tid=14, entry_tid=9478' 2024-08-31T22:23:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=106, tag_tid=14, entry_tid=9478' 2024-08-31T22:23:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=106, tag_tid=14, entry_tid=9478' 2024-08-31T22:23:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9478 2024-08-31T22:23:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 9478 2024-08-31T22:23:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:20.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=106, tag_tid=14, entry_tid=9574], [object_number=105, tag_tid=14, entry_tid=9573], [object_number=104, tag_tid=14, entry_tid=9572], [object_number=107, tag_tid=14, entry_tid=9571]]], state=connected] 2024-08-31T22:23:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=106, tag_tid=14, entry_tid=9574], [object_number=105, tag_tid=14, entry_tid=9573], [object_number=104, tag_tid=14, entry_tid=9572], [object_number=107, tag_tid=14, entry_tid=9571]]], state=connected] 2024-08-31T22:23:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=106, tag_tid=14, entry_tid=9574' 2024-08-31T22:23:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=106, tag_tid=14, entry_tid=9574' 2024-08-31T22:23:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=106, tag_tid=14, entry_tid=9574' '!=' 'object_number=106, tag_tid=14, entry_tid=9478' 2024-08-31T22:23:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=106, tag_tid=14, entry_tid=9574' = 'object_number=106, tag_tid=14, entry_tid=9478' 2024-08-31T22:23:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=106, tag_tid=14, entry_tid=9574' 2024-08-31T22:23:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=106, tag_tid=14, entry_tid=9574' 2024-08-31T22:23:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=106, tag_tid=14, entry_tid=9574' 2024-08-31T22:23:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9574 2024-08-31T22:23:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 9574 2024-08-31T22:23:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=9717], [object_number=104, tag_tid=14, entry_tid=9716], [object_number=107, tag_tid=14, entry_tid=9715], [object_number=106, tag_tid=14, entry_tid=9714]]], state=connected] 2024-08-31T22:23:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:21.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:21.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=9717], [object_number=104, tag_tid=14, entry_tid=9716], [object_number=107, tag_tid=14, entry_tid=9715], [object_number=106, tag_tid=14, entry_tid=9714]]], state=connected] 2024-08-31T22:23:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=105, tag_tid=14, entry_tid=9717' 2024-08-31T22:23:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=105, tag_tid=14, entry_tid=9717' 2024-08-31T22:23:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=9717' '!=' 'object_number=106, tag_tid=14, entry_tid=9574' 2024-08-31T22:23:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=9717' = 'object_number=106, tag_tid=14, entry_tid=9574' 2024-08-31T22:23:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=105, tag_tid=14, entry_tid=9717' 2024-08-31T22:23:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=9717' 2024-08-31T22:23:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=9717' 2024-08-31T22:23:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9717 2024-08-31T22:23:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 9717 2024-08-31T22:23:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=9829], [object_number=104, tag_tid=14, entry_tid=9828], [object_number=107, tag_tid=14, entry_tid=9827], [object_number=106, tag_tid=14, entry_tid=9826]]], state=connected] 2024-08-31T22:23:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=9829], [object_number=104, tag_tid=14, entry_tid=9828], [object_number=107, tag_tid=14, entry_tid=9827], [object_number=106, tag_tid=14, entry_tid=9826]]], state=connected] 2024-08-31T22:23:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=105, tag_tid=14, entry_tid=9829' 2024-08-31T22:23:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=105, tag_tid=14, entry_tid=9829' 2024-08-31T22:23:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=9829' '!=' 'object_number=105, tag_tid=14, entry_tid=9717' 2024-08-31T22:23:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=9829' = 'object_number=105, tag_tid=14, entry_tid=9717' 2024-08-31T22:23:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=105, tag_tid=14, entry_tid=9829' 2024-08-31T22:23:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=9829' 2024-08-31T22:23:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:21.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:21.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:21.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=9829' 2024-08-31T22:23:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9829 2024-08-31T22:23:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 9829 2024-08-31T22:23:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=9973], [object_number=104, tag_tid=14, entry_tid=9972], [object_number=107, tag_tid=14, entry_tid=9971], [object_number=106, tag_tid=14, entry_tid=9970]]], state=connected] 2024-08-31T22:23:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=9973], [object_number=104, tag_tid=14, entry_tid=9972], [object_number=107, tag_tid=14, entry_tid=9971], [object_number=106, tag_tid=14, entry_tid=9970]]], state=connected] 2024-08-31T22:23:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=105, tag_tid=14, entry_tid=9973' 2024-08-31T22:23:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=105, tag_tid=14, entry_tid=9973' 2024-08-31T22:23:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=9973' '!=' 'object_number=105, tag_tid=14, entry_tid=9829' 2024-08-31T22:23:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=9973' = 'object_number=105, tag_tid=14, entry_tid=9829' 2024-08-31T22:23:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=105, tag_tid=14, entry_tid=9973' 2024-08-31T22:23:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=9973' 2024-08-31T22:23:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=9973' 2024-08-31T22:23:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9973 2024-08-31T22:23:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 9973 2024-08-31T22:23:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:22.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:22.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:22.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:22.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=10065], [object_number=104, tag_tid=14, entry_tid=10064], [object_number=107, tag_tid=14, entry_tid=10063], [object_number=106, tag_tid=14, entry_tid=10062]]], state=connected] 2024-08-31T22:23:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:22.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=105, tag_tid=14, entry_tid=10065], [object_number=104, tag_tid=14, entry_tid=10064], [object_number=107, tag_tid=14, entry_tid=10063], [object_number=106, tag_tid=14, entry_tid=10062]]], state=connected] 2024-08-31T22:23:22.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=105, tag_tid=14, entry_tid=10065' 2024-08-31T22:23:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=105, tag_tid=14, entry_tid=10065' 2024-08-31T22:23:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=10065' '!=' 'object_number=105, tag_tid=14, entry_tid=9973' 2024-08-31T22:23:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=105, tag_tid=14, entry_tid=10065' = 'object_number=105, tag_tid=14, entry_tid=9973' 2024-08-31T22:23:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=105, tag_tid=14, entry_tid=10065' 2024-08-31T22:23:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=10065' 2024-08-31T22:23:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:22.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:22.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=105, tag_tid=14, entry_tid=10065' 2024-08-31T22:23:22.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:22.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10065 2024-08-31T22:23:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 10065 2024-08-31T22:23:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:23.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=107, tag_tid=14, entry_tid=10159], [object_number=106, tag_tid=14, entry_tid=10158], [object_number=105, tag_tid=14, entry_tid=10157], [object_number=104, tag_tid=14, entry_tid=10156]]], state=connected] 2024-08-31T22:23:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=107, tag_tid=14, entry_tid=10159], [object_number=106, tag_tid=14, entry_tid=10158], [object_number=105, tag_tid=14, entry_tid=10157], [object_number=104, tag_tid=14, entry_tid=10156]]], state=connected] 2024-08-31T22:23:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:23.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=107, tag_tid=14, entry_tid=10159' 2024-08-31T22:23:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=107, tag_tid=14, entry_tid=10159' 2024-08-31T22:23:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=107, tag_tid=14, entry_tid=10159' '!=' 'object_number=105, tag_tid=14, entry_tid=10065' 2024-08-31T22:23:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=107, tag_tid=14, entry_tid=10159' = 'object_number=105, tag_tid=14, entry_tid=10065' 2024-08-31T22:23:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=107, tag_tid=14, entry_tid=10159' 2024-08-31T22:23:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=107, tag_tid=14, entry_tid=10159' 2024-08-31T22:23:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=107, tag_tid=14, entry_tid=10159' 2024-08-31T22:23:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10159 2024-08-31T22:23:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 10159 2024-08-31T22:23:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=104, tag_tid=14, entry_tid=10236], [object_number=107, tag_tid=14, entry_tid=10235], [object_number=106, tag_tid=14, entry_tid=10234], [object_number=105, tag_tid=14, entry_tid=10233]]], state=connected] 2024-08-31T22:23:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 26 2024-08-31T22:23:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=104, tag_tid=14, entry_tid=10236], [object_number=107, tag_tid=14, entry_tid=10235], [object_number=106, tag_tid=14, entry_tid=10234], [object_number=105, tag_tid=14, entry_tid=10233]]], state=connected] 2024-08-31T22:23:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=104, tag_tid=14, entry_tid=10236' 2024-08-31T22:23:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=104, tag_tid=14, entry_tid=10236' 2024-08-31T22:23:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=104, tag_tid=14, entry_tid=10236' '!=' 'object_number=107, tag_tid=14, entry_tid=10159' 2024-08-31T22:23:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=104, tag_tid=14, entry_tid=10236' = 'object_number=107, tag_tid=14, entry_tid=10159' 2024-08-31T22:23:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=104, tag_tid=14, entry_tid=10236' 2024-08-31T22:23:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=104, tag_tid=14, entry_tid=10236' 2024-08-31T22:23:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=104, tag_tid=14, entry_tid=10236' 2024-08-31T22:23:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10236 2024-08-31T22:23:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 10236 2024-08-31T22:23:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=108, tag_tid=14, entry_tid=10276], [object_number=111, tag_tid=14, entry_tid=10275], [object_number=110, tag_tid=14, entry_tid=10274], [object_number=109, tag_tid=14, entry_tid=10273]]], state=connected] 2024-08-31T22:23:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=108, tag_tid=14, entry_tid=10276], [object_number=111, tag_tid=14, entry_tid=10275], [object_number=110, tag_tid=14, entry_tid=10274], [object_number=109, tag_tid=14, entry_tid=10273]]], state=connected] 2024-08-31T22:23:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=108, tag_tid=14, entry_tid=10276' 2024-08-31T22:23:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=108, tag_tid=14, entry_tid=10276' 2024-08-31T22:23:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=108, tag_tid=14, entry_tid=10276' '!=' 'object_number=104, tag_tid=14, entry_tid=10236' 2024-08-31T22:23:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=108, tag_tid=14, entry_tid=10276' = 'object_number=104, tag_tid=14, entry_tid=10236' 2024-08-31T22:23:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=108, tag_tid=14, entry_tid=10276' 2024-08-31T22:23:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:24.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=108, tag_tid=14, entry_tid=10276' 2024-08-31T22:23:24.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:24.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:24.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=108, tag_tid=14, entry_tid=10276' 2024-08-31T22:23:24.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:24.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:24.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10276 2024-08-31T22:23:24.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 10276 2024-08-31T22:23:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=111, tag_tid=14, entry_tid=10371], [object_number=110, tag_tid=14, entry_tid=10370], [object_number=109, tag_tid=14, entry_tid=10369], [object_number=108, tag_tid=14, entry_tid=10368]]], state=connected] 2024-08-31T22:23:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=111, tag_tid=14, entry_tid=10371], [object_number=110, tag_tid=14, entry_tid=10370], [object_number=109, tag_tid=14, entry_tid=10369], [object_number=108, tag_tid=14, entry_tid=10368]]], state=connected] 2024-08-31T22:23:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=111, tag_tid=14, entry_tid=10371' 2024-08-31T22:23:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=111, tag_tid=14, entry_tid=10371' 2024-08-31T22:23:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=111, tag_tid=14, entry_tid=10371' '!=' 'object_number=108, tag_tid=14, entry_tid=10276' 2024-08-31T22:23:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=111, tag_tid=14, entry_tid=10371' = 'object_number=108, tag_tid=14, entry_tid=10276' 2024-08-31T22:23:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=111, tag_tid=14, entry_tid=10371' 2024-08-31T22:23:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=111, tag_tid=14, entry_tid=10371' 2024-08-31T22:23:24.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:24.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=111, tag_tid=14, entry_tid=10371' 2024-08-31T22:23:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:24.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10371 2024-08-31T22:23:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 10371 2024-08-31T22:23:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:25.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=110, tag_tid=14, entry_tid=10470], [object_number=109, tag_tid=14, entry_tid=10469], [object_number=108, tag_tid=14, entry_tid=10468], [object_number=111, tag_tid=14, entry_tid=10467]]], state=connected] 2024-08-31T22:23:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=110, tag_tid=14, entry_tid=10470], [object_number=109, tag_tid=14, entry_tid=10469], [object_number=108, tag_tid=14, entry_tid=10468], [object_number=111, tag_tid=14, entry_tid=10467]]], state=connected] 2024-08-31T22:23:25.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=110, tag_tid=14, entry_tid=10470' 2024-08-31T22:23:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=110, tag_tid=14, entry_tid=10470' 2024-08-31T22:23:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=110, tag_tid=14, entry_tid=10470' '!=' 'object_number=111, tag_tid=14, entry_tid=10371' 2024-08-31T22:23:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=110, tag_tid=14, entry_tid=10470' = 'object_number=111, tag_tid=14, entry_tid=10371' 2024-08-31T22:23:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=110, tag_tid=14, entry_tid=10470' 2024-08-31T22:23:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=110, tag_tid=14, entry_tid=10470' 2024-08-31T22:23:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=110, tag_tid=14, entry_tid=10470' 2024-08-31T22:23:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10470 2024-08-31T22:23:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 10470 2024-08-31T22:23:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=110, tag_tid=14, entry_tid=10566], [object_number=109, tag_tid=14, entry_tid=10565], [object_number=108, tag_tid=14, entry_tid=10564], [object_number=111, tag_tid=14, entry_tid=10563]]], state=connected] 2024-08-31T22:23:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=110, tag_tid=14, entry_tid=10566], [object_number=109, tag_tid=14, entry_tid=10565], [object_number=108, tag_tid=14, entry_tid=10564], [object_number=111, tag_tid=14, entry_tid=10563]]], state=connected] 2024-08-31T22:23:25.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=110, tag_tid=14, entry_tid=10566' 2024-08-31T22:23:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=110, tag_tid=14, entry_tid=10566' 2024-08-31T22:23:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=110, tag_tid=14, entry_tid=10566' '!=' 'object_number=110, tag_tid=14, entry_tid=10470' 2024-08-31T22:23:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=110, tag_tid=14, entry_tid=10566' = 'object_number=110, tag_tid=14, entry_tid=10470' 2024-08-31T22:23:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=110, tag_tid=14, entry_tid=10566' 2024-08-31T22:23:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=110, tag_tid=14, entry_tid=10566' 2024-08-31T22:23:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=110, tag_tid=14, entry_tid=10566' 2024-08-31T22:23:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10566 2024-08-31T22:23:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 10566 2024-08-31T22:23:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:26.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:26.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:26.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:26.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:26.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:26.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:26.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:26.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=110, tag_tid=14, entry_tid=10654], [object_number=109, tag_tid=14, entry_tid=10653], [object_number=108, tag_tid=14, entry_tid=10652], [object_number=111, tag_tid=14, entry_tid=10651]]], state=connected] 2024-08-31T22:23:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=110, tag_tid=14, entry_tid=10654], [object_number=109, tag_tid=14, entry_tid=10653], [object_number=108, tag_tid=14, entry_tid=10652], [object_number=111, tag_tid=14, entry_tid=10651]]], state=connected] 2024-08-31T22:23:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=110, tag_tid=14, entry_tid=10654' 2024-08-31T22:23:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=110, tag_tid=14, entry_tid=10654' 2024-08-31T22:23:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=110, tag_tid=14, entry_tid=10654' '!=' 'object_number=110, tag_tid=14, entry_tid=10566' 2024-08-31T22:23:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=110, tag_tid=14, entry_tid=10654' = 'object_number=110, tag_tid=14, entry_tid=10566' 2024-08-31T22:23:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=110, tag_tid=14, entry_tid=10654' 2024-08-31T22:23:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:26.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=110, tag_tid=14, entry_tid=10654' 2024-08-31T22:23:26.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:26.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=110, tag_tid=14, entry_tid=10654' 2024-08-31T22:23:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10654 2024-08-31T22:23:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 10654 2024-08-31T22:23:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=10741], [object_number=108, tag_tid=14, entry_tid=10740], [object_number=111, tag_tid=14, entry_tid=10739], [object_number=110, tag_tid=14, entry_tid=10738]]], state=connected] 2024-08-31T22:23:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:26.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:26.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:26.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:26.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:26.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=10741], [object_number=108, tag_tid=14, entry_tid=10740], [object_number=111, tag_tid=14, entry_tid=10739], [object_number=110, tag_tid=14, entry_tid=10738]]], state=connected] 2024-08-31T22:23:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=109, tag_tid=14, entry_tid=10741' 2024-08-31T22:23:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=109, tag_tid=14, entry_tid=10741' 2024-08-31T22:23:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=10741' '!=' 'object_number=110, tag_tid=14, entry_tid=10654' 2024-08-31T22:23:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=10741' = 'object_number=110, tag_tid=14, entry_tid=10654' 2024-08-31T22:23:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=109, tag_tid=14, entry_tid=10741' 2024-08-31T22:23:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:26.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=10741' 2024-08-31T22:23:26.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:26.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=10741' 2024-08-31T22:23:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10741 2024-08-31T22:23:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 10741 2024-08-31T22:23:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=10833], [object_number=108, tag_tid=14, entry_tid=10832], [object_number=111, tag_tid=14, entry_tid=10831], [object_number=110, tag_tid=14, entry_tid=10830]]], state=connected] 2024-08-31T22:23:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=10833], [object_number=108, tag_tid=14, entry_tid=10832], [object_number=111, tag_tid=14, entry_tid=10831], [object_number=110, tag_tid=14, entry_tid=10830]]], state=connected] 2024-08-31T22:23:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=109, tag_tid=14, entry_tid=10833' 2024-08-31T22:23:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=109, tag_tid=14, entry_tid=10833' 2024-08-31T22:23:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=10833' '!=' 'object_number=109, tag_tid=14, entry_tid=10741' 2024-08-31T22:23:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=10833' = 'object_number=109, tag_tid=14, entry_tid=10741' 2024-08-31T22:23:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=109, tag_tid=14, entry_tid=10833' 2024-08-31T22:23:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=10833' 2024-08-31T22:23:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=10833' 2024-08-31T22:23:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10833 2024-08-31T22:23:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 10833 2024-08-31T22:23:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=110, tag_tid=14, entry_tid=10934], [object_number=109, tag_tid=14, entry_tid=10933], [object_number=108, tag_tid=14, entry_tid=10932], [object_number=111, tag_tid=14, entry_tid=10931]]], state=connected] 2024-08-31T22:23:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=110, tag_tid=14, entry_tid=10934], [object_number=109, tag_tid=14, entry_tid=10933], [object_number=108, tag_tid=14, entry_tid=10932], [object_number=111, tag_tid=14, entry_tid=10931]]], state=connected] 2024-08-31T22:23:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=110, tag_tid=14, entry_tid=10934' 2024-08-31T22:23:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=110, tag_tid=14, entry_tid=10934' 2024-08-31T22:23:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=110, tag_tid=14, entry_tid=10934' '!=' 'object_number=109, tag_tid=14, entry_tid=10833' 2024-08-31T22:23:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=110, tag_tid=14, entry_tid=10934' = 'object_number=109, tag_tid=14, entry_tid=10833' 2024-08-31T22:23:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=110, tag_tid=14, entry_tid=10934' 2024-08-31T22:23:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=110, tag_tid=14, entry_tid=10934' 2024-08-31T22:23:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=110, tag_tid=14, entry_tid=10934' 2024-08-31T22:23:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:27.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10934 2024-08-31T22:23:27.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 10934 2024-08-31T22:23:27.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:27.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:27.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:28.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=11033], [object_number=108, tag_tid=14, entry_tid=11032], [object_number=111, tag_tid=14, entry_tid=11031], [object_number=110, tag_tid=14, entry_tid=11030]]], state=connected] 2024-08-31T22:23:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:28.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:28.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=11033], [object_number=108, tag_tid=14, entry_tid=11032], [object_number=111, tag_tid=14, entry_tid=11031], [object_number=110, tag_tid=14, entry_tid=11030]]], state=connected] 2024-08-31T22:23:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=109, tag_tid=14, entry_tid=11033' 2024-08-31T22:23:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=109, tag_tid=14, entry_tid=11033' 2024-08-31T22:23:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=11033' '!=' 'object_number=110, tag_tid=14, entry_tid=10934' 2024-08-31T22:23:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=11033' = 'object_number=110, tag_tid=14, entry_tid=10934' 2024-08-31T22:23:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=109, tag_tid=14, entry_tid=11033' 2024-08-31T22:23:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=11033' 2024-08-31T22:23:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=11033' 2024-08-31T22:23:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11033 2024-08-31T22:23:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 11033 2024-08-31T22:23:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=110, tag_tid=14, entry_tid=11178], [object_number=109, tag_tid=14, entry_tid=11177], [object_number=108, tag_tid=14, entry_tid=11176], [object_number=111, tag_tid=14, entry_tid=11175]]], state=connected] 2024-08-31T22:23:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:28.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:28.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:28.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:28.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:28.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:28.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:28.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:28.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:28.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:28.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:28.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:28.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:28.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:28.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:28.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=110, tag_tid=14, entry_tid=11178], [object_number=109, tag_tid=14, entry_tid=11177], [object_number=108, tag_tid=14, entry_tid=11176], [object_number=111, tag_tid=14, entry_tid=11175]]], state=connected] 2024-08-31T22:23:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=110, tag_tid=14, entry_tid=11178' 2024-08-31T22:23:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=110, tag_tid=14, entry_tid=11178' 2024-08-31T22:23:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=110, tag_tid=14, entry_tid=11178' '!=' 'object_number=109, tag_tid=14, entry_tid=11033' 2024-08-31T22:23:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=110, tag_tid=14, entry_tid=11178' = 'object_number=109, tag_tid=14, entry_tid=11033' 2024-08-31T22:23:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=110, tag_tid=14, entry_tid=11178' 2024-08-31T22:23:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=110, tag_tid=14, entry_tid=11178' 2024-08-31T22:23:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=110, tag_tid=14, entry_tid=11178' 2024-08-31T22:23:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11178 2024-08-31T22:23:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 11178 2024-08-31T22:23:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:29.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:29.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:29.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:29.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:29.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:29.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=11269], [object_number=108, tag_tid=14, entry_tid=11268], [object_number=111, tag_tid=14, entry_tid=11267], [object_number=110, tag_tid=14, entry_tid=11266]]], state=connected] 2024-08-31T22:23:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=11269], [object_number=108, tag_tid=14, entry_tid=11268], [object_number=111, tag_tid=14, entry_tid=11267], [object_number=110, tag_tid=14, entry_tid=11266]]], state=connected] 2024-08-31T22:23:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=109, tag_tid=14, entry_tid=11269' 2024-08-31T22:23:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=109, tag_tid=14, entry_tid=11269' 2024-08-31T22:23:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=11269' '!=' 'object_number=110, tag_tid=14, entry_tid=11178' 2024-08-31T22:23:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=11269' = 'object_number=110, tag_tid=14, entry_tid=11178' 2024-08-31T22:23:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=109, tag_tid=14, entry_tid=11269' 2024-08-31T22:23:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:29.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=11269' 2024-08-31T22:23:29.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:29.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:29.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=11269' 2024-08-31T22:23:29.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:29.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11269 2024-08-31T22:23:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 11269 2024-08-31T22:23:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=11357], [object_number=108, tag_tid=14, entry_tid=11356], [object_number=111, tag_tid=14, entry_tid=11355], [object_number=110, tag_tid=14, entry_tid=11354]]], state=connected] 2024-08-31T22:23:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:29.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:29.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=11357], [object_number=108, tag_tid=14, entry_tid=11356], [object_number=111, tag_tid=14, entry_tid=11355], [object_number=110, tag_tid=14, entry_tid=11354]]], state=connected] 2024-08-31T22:23:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=109, tag_tid=14, entry_tid=11357' 2024-08-31T22:23:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=109, tag_tid=14, entry_tid=11357' 2024-08-31T22:23:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=11357' '!=' 'object_number=109, tag_tid=14, entry_tid=11269' 2024-08-31T22:23:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=11357' = 'object_number=109, tag_tid=14, entry_tid=11269' 2024-08-31T22:23:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=109, tag_tid=14, entry_tid=11357' 2024-08-31T22:23:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=11357' 2024-08-31T22:23:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=11357' 2024-08-31T22:23:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11357 2024-08-31T22:23:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 11357 2024-08-31T22:23:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:30.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:30.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:30.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:30.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:30.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:30.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:30.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:30.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:30.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=108, tag_tid=14, entry_tid=11440], [object_number=111, tag_tid=14, entry_tid=11439], [object_number=110, tag_tid=14, entry_tid=11438], [object_number=109, tag_tid=14, entry_tid=11437]]], state=connected] 2024-08-31T22:23:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=108, tag_tid=14, entry_tid=11440], [object_number=111, tag_tid=14, entry_tid=11439], [object_number=110, tag_tid=14, entry_tid=11438], [object_number=109, tag_tid=14, entry_tid=11437]]], state=connected] 2024-08-31T22:23:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=108, tag_tid=14, entry_tid=11440' 2024-08-31T22:23:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=108, tag_tid=14, entry_tid=11440' 2024-08-31T22:23:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=108, tag_tid=14, entry_tid=11440' '!=' 'object_number=109, tag_tid=14, entry_tid=11357' 2024-08-31T22:23:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=108, tag_tid=14, entry_tid=11440' = 'object_number=109, tag_tid=14, entry_tid=11357' 2024-08-31T22:23:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=108, tag_tid=14, entry_tid=11440' 2024-08-31T22:23:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=108, tag_tid=14, entry_tid=11440' 2024-08-31T22:23:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=108, tag_tid=14, entry_tid=11440' 2024-08-31T22:23:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11440 2024-08-31T22:23:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 11440 2024-08-31T22:23:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:30.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=110, tag_tid=14, entry_tid=11526], [object_number=109, tag_tid=14, entry_tid=11525], [object_number=108, tag_tid=14, entry_tid=11524], [object_number=111, tag_tid=14, entry_tid=11523]]], state=connected] 2024-08-31T22:23:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=110, tag_tid=14, entry_tid=11526], [object_number=109, tag_tid=14, entry_tid=11525], [object_number=108, tag_tid=14, entry_tid=11524], [object_number=111, tag_tid=14, entry_tid=11523]]], state=connected] 2024-08-31T22:23:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=110, tag_tid=14, entry_tid=11526' 2024-08-31T22:23:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=110, tag_tid=14, entry_tid=11526' 2024-08-31T22:23:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=110, tag_tid=14, entry_tid=11526' '!=' 'object_number=108, tag_tid=14, entry_tid=11440' 2024-08-31T22:23:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=110, tag_tid=14, entry_tid=11526' = 'object_number=108, tag_tid=14, entry_tid=11440' 2024-08-31T22:23:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=110, tag_tid=14, entry_tid=11526' 2024-08-31T22:23:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=110, tag_tid=14, entry_tid=11526' 2024-08-31T22:23:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=110, tag_tid=14, entry_tid=11526' 2024-08-31T22:23:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11526 2024-08-31T22:23:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 11526 2024-08-31T22:23:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=108, tag_tid=14, entry_tid=11612], [object_number=111, tag_tid=14, entry_tid=11611], [object_number=110, tag_tid=14, entry_tid=11610], [object_number=109, tag_tid=14, entry_tid=11609]]], state=connected] 2024-08-31T22:23:31.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=108, tag_tid=14, entry_tid=11612], [object_number=111, tag_tid=14, entry_tid=11611], [object_number=110, tag_tid=14, entry_tid=11610], [object_number=109, tag_tid=14, entry_tid=11609]]], state=connected] 2024-08-31T22:23:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=108, tag_tid=14, entry_tid=11612' 2024-08-31T22:23:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=108, tag_tid=14, entry_tid=11612' 2024-08-31T22:23:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=108, tag_tid=14, entry_tid=11612' '!=' 'object_number=110, tag_tid=14, entry_tid=11526' 2024-08-31T22:23:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=108, tag_tid=14, entry_tid=11612' = 'object_number=110, tag_tid=14, entry_tid=11526' 2024-08-31T22:23:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=108, tag_tid=14, entry_tid=11612' 2024-08-31T22:23:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=108, tag_tid=14, entry_tid=11612' 2024-08-31T22:23:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=108, tag_tid=14, entry_tid=11612' 2024-08-31T22:23:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11612 2024-08-31T22:23:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 11612 2024-08-31T22:23:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=110, tag_tid=14, entry_tid=11718], [object_number=109, tag_tid=14, entry_tid=11717], [object_number=108, tag_tid=14, entry_tid=11716], [object_number=111, tag_tid=14, entry_tid=11715]]], state=connected] 2024-08-31T22:23:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=110, tag_tid=14, entry_tid=11718], [object_number=109, tag_tid=14, entry_tid=11717], [object_number=108, tag_tid=14, entry_tid=11716], [object_number=111, tag_tid=14, entry_tid=11715]]], state=connected] 2024-08-31T22:23:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=110, tag_tid=14, entry_tid=11718' 2024-08-31T22:23:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=110, tag_tid=14, entry_tid=11718' 2024-08-31T22:23:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=110, tag_tid=14, entry_tid=11718' '!=' 'object_number=108, tag_tid=14, entry_tid=11612' 2024-08-31T22:23:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=110, tag_tid=14, entry_tid=11718' = 'object_number=108, tag_tid=14, entry_tid=11612' 2024-08-31T22:23:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=110, tag_tid=14, entry_tid=11718' 2024-08-31T22:23:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=110, tag_tid=14, entry_tid=11718' 2024-08-31T22:23:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=110, tag_tid=14, entry_tid=11718' 2024-08-31T22:23:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11718 2024-08-31T22:23:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 11718 2024-08-31T22:23:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:32.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:32.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:32.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:32.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:32.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=11817], [object_number=108, tag_tid=14, entry_tid=11816], [object_number=111, tag_tid=14, entry_tid=11815], [object_number=110, tag_tid=14, entry_tid=11814]]], state=connected] 2024-08-31T22:23:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=11817], [object_number=108, tag_tid=14, entry_tid=11816], [object_number=111, tag_tid=14, entry_tid=11815], [object_number=110, tag_tid=14, entry_tid=11814]]], state=connected] 2024-08-31T22:23:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=109, tag_tid=14, entry_tid=11817' 2024-08-31T22:23:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=109, tag_tid=14, entry_tid=11817' 2024-08-31T22:23:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=11817' '!=' 'object_number=110, tag_tid=14, entry_tid=11718' 2024-08-31T22:23:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=11817' = 'object_number=110, tag_tid=14, entry_tid=11718' 2024-08-31T22:23:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=109, tag_tid=14, entry_tid=11817' 2024-08-31T22:23:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=11817' 2024-08-31T22:23:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=11817' 2024-08-31T22:23:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11817 2024-08-31T22:23:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 11817 2024-08-31T22:23:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=11913], [object_number=108, tag_tid=14, entry_tid=11912], [object_number=111, tag_tid=14, entry_tid=11911], [object_number=110, tag_tid=14, entry_tid=11910]]], state=connected] 2024-08-31T22:23:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=11913], [object_number=108, tag_tid=14, entry_tid=11912], [object_number=111, tag_tid=14, entry_tid=11911], [object_number=110, tag_tid=14, entry_tid=11910]]], state=connected] 2024-08-31T22:23:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=109, tag_tid=14, entry_tid=11913' 2024-08-31T22:23:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=109, tag_tid=14, entry_tid=11913' 2024-08-31T22:23:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=11913' '!=' 'object_number=109, tag_tid=14, entry_tid=11817' 2024-08-31T22:23:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=11913' = 'object_number=109, tag_tid=14, entry_tid=11817' 2024-08-31T22:23:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=109, tag_tid=14, entry_tid=11913' 2024-08-31T22:23:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=11913' 2024-08-31T22:23:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:32.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:32.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:32.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:32.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=11913' 2024-08-31T22:23:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11913 2024-08-31T22:23:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 11913 2024-08-31T22:23:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=111, tag_tid=14, entry_tid=11999], [object_number=110, tag_tid=14, entry_tid=11998], [object_number=109, tag_tid=14, entry_tid=11997], [object_number=108, tag_tid=14, entry_tid=11996]]], state=connected] 2024-08-31T22:23:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=111, tag_tid=14, entry_tid=11999], [object_number=110, tag_tid=14, entry_tid=11998], [object_number=109, tag_tid=14, entry_tid=11997], [object_number=108, tag_tid=14, entry_tid=11996]]], state=connected] 2024-08-31T22:23:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=111, tag_tid=14, entry_tid=11999' 2024-08-31T22:23:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=111, tag_tid=14, entry_tid=11999' 2024-08-31T22:23:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=111, tag_tid=14, entry_tid=11999' '!=' 'object_number=109, tag_tid=14, entry_tid=11913' 2024-08-31T22:23:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=111, tag_tid=14, entry_tid=11999' = 'object_number=109, tag_tid=14, entry_tid=11913' 2024-08-31T22:23:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=111, tag_tid=14, entry_tid=11999' 2024-08-31T22:23:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=111, tag_tid=14, entry_tid=11999' 2024-08-31T22:23:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=111, tag_tid=14, entry_tid=11999' 2024-08-31T22:23:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11999 2024-08-31T22:23:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 11999 2024-08-31T22:23:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:33.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:33.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=111, tag_tid=14, entry_tid=12083], [object_number=110, tag_tid=14, entry_tid=12082], [object_number=109, tag_tid=14, entry_tid=12081], [object_number=108, tag_tid=14, entry_tid=12080]]], state=connected] 2024-08-31T22:23:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=111, tag_tid=14, entry_tid=12083], [object_number=110, tag_tid=14, entry_tid=12082], [object_number=109, tag_tid=14, entry_tid=12081], [object_number=108, tag_tid=14, entry_tid=12080]]], state=connected] 2024-08-31T22:23:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=111, tag_tid=14, entry_tid=12083' 2024-08-31T22:23:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=111, tag_tid=14, entry_tid=12083' 2024-08-31T22:23:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=111, tag_tid=14, entry_tid=12083' '!=' 'object_number=111, tag_tid=14, entry_tid=11999' 2024-08-31T22:23:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=111, tag_tid=14, entry_tid=12083' = 'object_number=111, tag_tid=14, entry_tid=11999' 2024-08-31T22:23:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=111, tag_tid=14, entry_tid=12083' 2024-08-31T22:23:33.958 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:23:33.954+0000 7f1a77b11640 -1 reset not still connected to 0x5649bd57d5f0 2024-08-31T22:23:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=111, tag_tid=14, entry_tid=12083' 2024-08-31T22:23:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=111, tag_tid=14, entry_tid=12083' 2024-08-31T22:23:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12083 2024-08-31T22:23:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 12083 2024-08-31T22:23:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=111, tag_tid=14, entry_tid=12167], [object_number=110, tag_tid=14, entry_tid=12166], [object_number=109, tag_tid=14, entry_tid=12165], [object_number=108, tag_tid=14, entry_tid=12164]]], state=connected] 2024-08-31T22:23:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=111, tag_tid=14, entry_tid=12167], [object_number=110, tag_tid=14, entry_tid=12166], [object_number=109, tag_tid=14, entry_tid=12165], [object_number=108, tag_tid=14, entry_tid=12164]]], state=connected] 2024-08-31T22:23:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=111, tag_tid=14, entry_tid=12167' 2024-08-31T22:23:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=111, tag_tid=14, entry_tid=12167' 2024-08-31T22:23:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=111, tag_tid=14, entry_tid=12167' '!=' 'object_number=111, tag_tid=14, entry_tid=12083' 2024-08-31T22:23:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=111, tag_tid=14, entry_tid=12167' = 'object_number=111, tag_tid=14, entry_tid=12083' 2024-08-31T22:23:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=111, tag_tid=14, entry_tid=12167' 2024-08-31T22:23:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=111, tag_tid=14, entry_tid=12167' 2024-08-31T22:23:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:34.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=111, tag_tid=14, entry_tid=12167' 2024-08-31T22:23:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12167 2024-08-31T22:23:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 12167 2024-08-31T22:23:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=111, tag_tid=14, entry_tid=12247], [object_number=110, tag_tid=14, entry_tid=12246], [object_number=109, tag_tid=14, entry_tid=12245], [object_number=108, tag_tid=14, entry_tid=12244]]], state=connected] 2024-08-31T22:23:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=111, tag_tid=14, entry_tid=12247], [object_number=110, tag_tid=14, entry_tid=12246], [object_number=109, tag_tid=14, entry_tid=12245], [object_number=108, tag_tid=14, entry_tid=12244]]], state=connected] 2024-08-31T22:23:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=111, tag_tid=14, entry_tid=12247' 2024-08-31T22:23:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=111, tag_tid=14, entry_tid=12247' 2024-08-31T22:23:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=111, tag_tid=14, entry_tid=12247' '!=' 'object_number=111, tag_tid=14, entry_tid=12167' 2024-08-31T22:23:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=111, tag_tid=14, entry_tid=12247' = 'object_number=111, tag_tid=14, entry_tid=12167' 2024-08-31T22:23:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=111, tag_tid=14, entry_tid=12247' 2024-08-31T22:23:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=111, tag_tid=14, entry_tid=12247' 2024-08-31T22:23:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=111, tag_tid=14, entry_tid=12247' 2024-08-31T22:23:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12247 2024-08-31T22:23:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 12247 2024-08-31T22:23:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=110, tag_tid=14, entry_tid=12326], [object_number=109, tag_tid=14, entry_tid=12325], [object_number=108, tag_tid=14, entry_tid=12324], [object_number=111, tag_tid=14, entry_tid=12323]]], state=connected] 2024-08-31T22:23:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=110, tag_tid=14, entry_tid=12326], [object_number=109, tag_tid=14, entry_tid=12325], [object_number=108, tag_tid=14, entry_tid=12324], [object_number=111, tag_tid=14, entry_tid=12323]]], state=connected] 2024-08-31T22:23:35.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=110, tag_tid=14, entry_tid=12326' 2024-08-31T22:23:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=110, tag_tid=14, entry_tid=12326' 2024-08-31T22:23:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=110, tag_tid=14, entry_tid=12326' '!=' 'object_number=111, tag_tid=14, entry_tid=12247' 2024-08-31T22:23:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=110, tag_tid=14, entry_tid=12326' = 'object_number=111, tag_tid=14, entry_tid=12247' 2024-08-31T22:23:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=110, tag_tid=14, entry_tid=12326' 2024-08-31T22:23:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=110, tag_tid=14, entry_tid=12326' 2024-08-31T22:23:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=110, tag_tid=14, entry_tid=12326' 2024-08-31T22:23:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12326 2024-08-31T22:23:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 12326 2024-08-31T22:23:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=12413], [object_number=108, tag_tid=14, entry_tid=12412], [object_number=111, tag_tid=14, entry_tid=12411], [object_number=110, tag_tid=14, entry_tid=12410]]], state=connected] 2024-08-31T22:23:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=12413], [object_number=108, tag_tid=14, entry_tid=12412], [object_number=111, tag_tid=14, entry_tid=12411], [object_number=110, tag_tid=14, entry_tid=12410]]], state=connected] 2024-08-31T22:23:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=109, tag_tid=14, entry_tid=12413' 2024-08-31T22:23:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=109, tag_tid=14, entry_tid=12413' 2024-08-31T22:23:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=12413' '!=' 'object_number=110, tag_tid=14, entry_tid=12326' 2024-08-31T22:23:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=12413' = 'object_number=110, tag_tid=14, entry_tid=12326' 2024-08-31T22:23:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=109, tag_tid=14, entry_tid=12413' 2024-08-31T22:23:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=12413' 2024-08-31T22:23:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=12413' 2024-08-31T22:23:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12413 2024-08-31T22:23:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 12413 2024-08-31T22:23:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=111, tag_tid=14, entry_tid=12507], [object_number=110, tag_tid=14, entry_tid=12506], [object_number=109, tag_tid=14, entry_tid=12505], [object_number=108, tag_tid=14, entry_tid=12504]]], state=connected] 2024-08-31T22:23:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=111, tag_tid=14, entry_tid=12507], [object_number=110, tag_tid=14, entry_tid=12506], [object_number=109, tag_tid=14, entry_tid=12505], [object_number=108, tag_tid=14, entry_tid=12504]]], state=connected] 2024-08-31T22:23:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=111, tag_tid=14, entry_tid=12507' 2024-08-31T22:23:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=111, tag_tid=14, entry_tid=12507' 2024-08-31T22:23:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=111, tag_tid=14, entry_tid=12507' '!=' 'object_number=109, tag_tid=14, entry_tid=12413' 2024-08-31T22:23:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=111, tag_tid=14, entry_tid=12507' = 'object_number=109, tag_tid=14, entry_tid=12413' 2024-08-31T22:23:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=111, tag_tid=14, entry_tid=12507' 2024-08-31T22:23:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=111, tag_tid=14, entry_tid=12507' 2024-08-31T22:23:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=111, tag_tid=14, entry_tid=12507' 2024-08-31T22:23:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12507 2024-08-31T22:23:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 12507 2024-08-31T22:23:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=108, tag_tid=14, entry_tid=12588], [object_number=111, tag_tid=14, entry_tid=12587], [object_number=110, tag_tid=14, entry_tid=12586], [object_number=109, tag_tid=14, entry_tid=12585]]], state=connected] 2024-08-31T22:23:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=108, tag_tid=14, entry_tid=12588], [object_number=111, tag_tid=14, entry_tid=12587], [object_number=110, tag_tid=14, entry_tid=12586], [object_number=109, tag_tid=14, entry_tid=12585]]], state=connected] 2024-08-31T22:23:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=108, tag_tid=14, entry_tid=12588' 2024-08-31T22:23:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=108, tag_tid=14, entry_tid=12588' 2024-08-31T22:23:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=108, tag_tid=14, entry_tid=12588' '!=' 'object_number=111, tag_tid=14, entry_tid=12507' 2024-08-31T22:23:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=108, tag_tid=14, entry_tid=12588' = 'object_number=111, tag_tid=14, entry_tid=12507' 2024-08-31T22:23:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=108, tag_tid=14, entry_tid=12588' 2024-08-31T22:23:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=108, tag_tid=14, entry_tid=12588' 2024-08-31T22:23:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=108, tag_tid=14, entry_tid=12588' 2024-08-31T22:23:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12588 2024-08-31T22:23:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 12588 2024-08-31T22:23:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=110, tag_tid=14, entry_tid=12674], [object_number=109, tag_tid=14, entry_tid=12673], [object_number=108, tag_tid=14, entry_tid=12672], [object_number=111, tag_tid=14, entry_tid=12671]]], state=connected] 2024-08-31T22:23:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=110, tag_tid=14, entry_tid=12674], [object_number=109, tag_tid=14, entry_tid=12673], [object_number=108, tag_tid=14, entry_tid=12672], [object_number=111, tag_tid=14, entry_tid=12671]]], state=connected] 2024-08-31T22:23:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=110, tag_tid=14, entry_tid=12674' 2024-08-31T22:23:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=110, tag_tid=14, entry_tid=12674' 2024-08-31T22:23:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=110, tag_tid=14, entry_tid=12674' '!=' 'object_number=108, tag_tid=14, entry_tid=12588' 2024-08-31T22:23:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=110, tag_tid=14, entry_tid=12674' = 'object_number=108, tag_tid=14, entry_tid=12588' 2024-08-31T22:23:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=110, tag_tid=14, entry_tid=12674' 2024-08-31T22:23:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=110, tag_tid=14, entry_tid=12674' 2024-08-31T22:23:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=110, tag_tid=14, entry_tid=12674' 2024-08-31T22:23:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12674 2024-08-31T22:23:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 12674 2024-08-31T22:23:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:37.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=12765], [object_number=108, tag_tid=14, entry_tid=12764], [object_number=111, tag_tid=14, entry_tid=12763], [object_number=110, tag_tid=14, entry_tid=12762]]], state=connected] 2024-08-31T22:23:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=12765], [object_number=108, tag_tid=14, entry_tid=12764], [object_number=111, tag_tid=14, entry_tid=12763], [object_number=110, tag_tid=14, entry_tid=12762]]], state=connected] 2024-08-31T22:23:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=109, tag_tid=14, entry_tid=12765' 2024-08-31T22:23:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=109, tag_tid=14, entry_tid=12765' 2024-08-31T22:23:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=12765' '!=' 'object_number=110, tag_tid=14, entry_tid=12674' 2024-08-31T22:23:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=12765' = 'object_number=110, tag_tid=14, entry_tid=12674' 2024-08-31T22:23:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=109, tag_tid=14, entry_tid=12765' 2024-08-31T22:23:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=12765' 2024-08-31T22:23:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=12765' 2024-08-31T22:23:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12765 2024-08-31T22:23:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 12765 2024-08-31T22:23:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=12853], [object_number=108, tag_tid=14, entry_tid=12852], [object_number=111, tag_tid=14, entry_tid=12851], [object_number=110, tag_tid=14, entry_tid=12850]]], state=connected] 2024-08-31T22:23:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=12853], [object_number=108, tag_tid=14, entry_tid=12852], [object_number=111, tag_tid=14, entry_tid=12851], [object_number=110, tag_tid=14, entry_tid=12850]]], state=connected] 2024-08-31T22:23:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=109, tag_tid=14, entry_tid=12853' 2024-08-31T22:23:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=109, tag_tid=14, entry_tid=12853' 2024-08-31T22:23:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=12853' '!=' 'object_number=109, tag_tid=14, entry_tid=12765' 2024-08-31T22:23:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=12853' = 'object_number=109, tag_tid=14, entry_tid=12765' 2024-08-31T22:23:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=109, tag_tid=14, entry_tid=12853' 2024-08-31T22:23:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=12853' 2024-08-31T22:23:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=12853' 2024-08-31T22:23:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12853 2024-08-31T22:23:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 12853 2024-08-31T22:23:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:38.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=108, tag_tid=14, entry_tid=12944], [object_number=111, tag_tid=14, entry_tid=12943], [object_number=110, tag_tid=14, entry_tid=12942], [object_number=109, tag_tid=14, entry_tid=12941]]], state=connected] 2024-08-31T22:23:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=108, tag_tid=14, entry_tid=12944], [object_number=111, tag_tid=14, entry_tid=12943], [object_number=110, tag_tid=14, entry_tid=12942], [object_number=109, tag_tid=14, entry_tid=12941]]], state=connected] 2024-08-31T22:23:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=108, tag_tid=14, entry_tid=12944' 2024-08-31T22:23:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=108, tag_tid=14, entry_tid=12944' 2024-08-31T22:23:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=108, tag_tid=14, entry_tid=12944' '!=' 'object_number=109, tag_tid=14, entry_tid=12853' 2024-08-31T22:23:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=108, tag_tid=14, entry_tid=12944' = 'object_number=109, tag_tid=14, entry_tid=12853' 2024-08-31T22:23:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=108, tag_tid=14, entry_tid=12944' 2024-08-31T22:23:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=108, tag_tid=14, entry_tid=12944' 2024-08-31T22:23:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=108, tag_tid=14, entry_tid=12944' 2024-08-31T22:23:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12944 2024-08-31T22:23:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 12944 2024-08-31T22:23:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:39.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=13033], [object_number=108, tag_tid=14, entry_tid=13032], [object_number=111, tag_tid=14, entry_tid=13031], [object_number=110, tag_tid=14, entry_tid=13030]]], state=connected] 2024-08-31T22:23:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=13033], [object_number=108, tag_tid=14, entry_tid=13032], [object_number=111, tag_tid=14, entry_tid=13031], [object_number=110, tag_tid=14, entry_tid=13030]]], state=connected] 2024-08-31T22:23:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=109, tag_tid=14, entry_tid=13033' 2024-08-31T22:23:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=109, tag_tid=14, entry_tid=13033' 2024-08-31T22:23:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=13033' '!=' 'object_number=108, tag_tid=14, entry_tid=12944' 2024-08-31T22:23:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=13033' = 'object_number=108, tag_tid=14, entry_tid=12944' 2024-08-31T22:23:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=109, tag_tid=14, entry_tid=13033' 2024-08-31T22:23:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=13033' 2024-08-31T22:23:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=13033' 2024-08-31T22:23:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13033 2024-08-31T22:23:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 13033 2024-08-31T22:23:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:39.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=108, tag_tid=14, entry_tid=13128], [object_number=111, tag_tid=14, entry_tid=13127], [object_number=110, tag_tid=14, entry_tid=13126], [object_number=109, tag_tid=14, entry_tid=13125]]], state=connected] 2024-08-31T22:23:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=108, tag_tid=14, entry_tid=13128], [object_number=111, tag_tid=14, entry_tid=13127], [object_number=110, tag_tid=14, entry_tid=13126], [object_number=109, tag_tid=14, entry_tid=13125]]], state=connected] 2024-08-31T22:23:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=108, tag_tid=14, entry_tid=13128' 2024-08-31T22:23:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=108, tag_tid=14, entry_tid=13128' 2024-08-31T22:23:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=108, tag_tid=14, entry_tid=13128' '!=' 'object_number=109, tag_tid=14, entry_tid=13033' 2024-08-31T22:23:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=108, tag_tid=14, entry_tid=13128' = 'object_number=109, tag_tid=14, entry_tid=13033' 2024-08-31T22:23:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=108, tag_tid=14, entry_tid=13128' 2024-08-31T22:23:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=108, tag_tid=14, entry_tid=13128' 2024-08-31T22:23:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=108, tag_tid=14, entry_tid=13128' 2024-08-31T22:23:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13128 2024-08-31T22:23:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 13128 2024-08-31T22:23:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=13225], [object_number=108, tag_tid=14, entry_tid=13224], [object_number=111, tag_tid=14, entry_tid=13223], [object_number=110, tag_tid=14, entry_tid=13222]]], state=connected] 2024-08-31T22:23:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:40.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:40.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=13225], [object_number=108, tag_tid=14, entry_tid=13224], [object_number=111, tag_tid=14, entry_tid=13223], [object_number=110, tag_tid=14, entry_tid=13222]]], state=connected] 2024-08-31T22:23:40.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:40.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=109, tag_tid=14, entry_tid=13225' 2024-08-31T22:23:40.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=109, tag_tid=14, entry_tid=13225' 2024-08-31T22:23:40.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=13225' '!=' 'object_number=108, tag_tid=14, entry_tid=13128' 2024-08-31T22:23:40.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:40.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=13225' = 'object_number=108, tag_tid=14, entry_tid=13128' 2024-08-31T22:23:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=109, tag_tid=14, entry_tid=13225' 2024-08-31T22:23:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=13225' 2024-08-31T22:23:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:40.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=13225' 2024-08-31T22:23:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13225 2024-08-31T22:23:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 13225 2024-08-31T22:23:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=13333], [object_number=108, tag_tid=14, entry_tid=13332], [object_number=111, tag_tid=14, entry_tid=13331], [object_number=110, tag_tid=14, entry_tid=13330]]], state=connected] 2024-08-31T22:23:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=109, tag_tid=14, entry_tid=13333], [object_number=108, tag_tid=14, entry_tid=13332], [object_number=111, tag_tid=14, entry_tid=13331], [object_number=110, tag_tid=14, entry_tid=13330]]], state=connected] 2024-08-31T22:23:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=109, tag_tid=14, entry_tid=13333' 2024-08-31T22:23:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=109, tag_tid=14, entry_tid=13333' 2024-08-31T22:23:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=13333' '!=' 'object_number=109, tag_tid=14, entry_tid=13225' 2024-08-31T22:23:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=109, tag_tid=14, entry_tid=13333' = 'object_number=109, tag_tid=14, entry_tid=13225' 2024-08-31T22:23:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=109, tag_tid=14, entry_tid=13333' 2024-08-31T22:23:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=13333' 2024-08-31T22:23:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=109, tag_tid=14, entry_tid=13333' 2024-08-31T22:23:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13333 2024-08-31T22:23:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 13333 2024-08-31T22:23:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:41.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:41.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:41.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:41.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:41.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:41.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:41.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=111, tag_tid=14, entry_tid=13423], [object_number=110, tag_tid=14, entry_tid=13422], [object_number=109, tag_tid=14, entry_tid=13421], [object_number=108, tag_tid=14, entry_tid=13420]]], state=connected] 2024-08-31T22:23:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:41.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=111, tag_tid=14, entry_tid=13423], [object_number=110, tag_tid=14, entry_tid=13422], [object_number=109, tag_tid=14, entry_tid=13421], [object_number=108, tag_tid=14, entry_tid=13420]]], state=connected] 2024-08-31T22:23:41.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=111, tag_tid=14, entry_tid=13423' 2024-08-31T22:23:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=111, tag_tid=14, entry_tid=13423' 2024-08-31T22:23:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=111, tag_tid=14, entry_tid=13423' '!=' 'object_number=109, tag_tid=14, entry_tid=13333' 2024-08-31T22:23:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=111, tag_tid=14, entry_tid=13423' = 'object_number=109, tag_tid=14, entry_tid=13333' 2024-08-31T22:23:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=111, tag_tid=14, entry_tid=13423' 2024-08-31T22:23:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=111, tag_tid=14, entry_tid=13423' 2024-08-31T22:23:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:41.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=111, tag_tid=14, entry_tid=13423' 2024-08-31T22:23:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13423 2024-08-31T22:23:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 13423 2024-08-31T22:23:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=108, tag_tid=14, entry_tid=13520], [object_number=111, tag_tid=14, entry_tid=13519], [object_number=110, tag_tid=14, entry_tid=13518], [object_number=109, tag_tid=14, entry_tid=13517]]], state=connected] 2024-08-31T22:23:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:41.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:41.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:41.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:41.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:41.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=108, tag_tid=14, entry_tid=13520], [object_number=111, tag_tid=14, entry_tid=13519], [object_number=110, tag_tid=14, entry_tid=13518], [object_number=109, tag_tid=14, entry_tid=13517]]], state=connected] 2024-08-31T22:23:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=108, tag_tid=14, entry_tid=13520' 2024-08-31T22:23:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=108, tag_tid=14, entry_tid=13520' 2024-08-31T22:23:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=108, tag_tid=14, entry_tid=13520' '!=' 'object_number=111, tag_tid=14, entry_tid=13423' 2024-08-31T22:23:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=108, tag_tid=14, entry_tid=13520' = 'object_number=111, tag_tid=14, entry_tid=13423' 2024-08-31T22:23:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=108, tag_tid=14, entry_tid=13520' 2024-08-31T22:23:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=108, tag_tid=14, entry_tid=13520' 2024-08-31T22:23:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=108, tag_tid=14, entry_tid=13520' 2024-08-31T22:23:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13520 2024-08-31T22:23:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 13520 2024-08-31T22:23:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:42.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:42.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:42.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:42.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:42.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:42.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=108, tag_tid=14, entry_tid=13608], [object_number=111, tag_tid=14, entry_tid=13607], [object_number=110, tag_tid=14, entry_tid=13606], [object_number=109, tag_tid=14, entry_tid=13605]]], state=connected] 2024-08-31T22:23:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=108, tag_tid=14, entry_tid=13608], [object_number=111, tag_tid=14, entry_tid=13607], [object_number=110, tag_tid=14, entry_tid=13606], [object_number=109, tag_tid=14, entry_tid=13605]]], state=connected] 2024-08-31T22:23:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=108, tag_tid=14, entry_tid=13608' 2024-08-31T22:23:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=108, tag_tid=14, entry_tid=13608' 2024-08-31T22:23:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=108, tag_tid=14, entry_tid=13608' '!=' 'object_number=108, tag_tid=14, entry_tid=13520' 2024-08-31T22:23:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=108, tag_tid=14, entry_tid=13608' = 'object_number=108, tag_tid=14, entry_tid=13520' 2024-08-31T22:23:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=108, tag_tid=14, entry_tid=13608' 2024-08-31T22:23:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=108, tag_tid=14, entry_tid=13608' 2024-08-31T22:23:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=108, tag_tid=14, entry_tid=13608' 2024-08-31T22:23:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13608 2024-08-31T22:23:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 13608 2024-08-31T22:23:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:42.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:42.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:42.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:42.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=111, tag_tid=14, entry_tid=13699], [object_number=110, tag_tid=14, entry_tid=13698], [object_number=109, tag_tid=14, entry_tid=13697], [object_number=108, tag_tid=14, entry_tid=13696]]], state=connected] 2024-08-31T22:23:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=111, tag_tid=14, entry_tid=13699], [object_number=110, tag_tid=14, entry_tid=13698], [object_number=109, tag_tid=14, entry_tid=13697], [object_number=108, tag_tid=14, entry_tid=13696]]], state=connected] 2024-08-31T22:23:42.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:42.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=111, tag_tid=14, entry_tid=13699' 2024-08-31T22:23:42.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=111, tag_tid=14, entry_tid=13699' 2024-08-31T22:23:42.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=111, tag_tid=14, entry_tid=13699' '!=' 'object_number=108, tag_tid=14, entry_tid=13608' 2024-08-31T22:23:42.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:42.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=111, tag_tid=14, entry_tid=13699' = 'object_number=108, tag_tid=14, entry_tid=13608' 2024-08-31T22:23:42.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=111, tag_tid=14, entry_tid=13699' 2024-08-31T22:23:42.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=111, tag_tid=14, entry_tid=13699' 2024-08-31T22:23:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=111, tag_tid=14, entry_tid=13699' 2024-08-31T22:23:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13699 2024-08-31T22:23:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 13699 2024-08-31T22:23:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:43.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:43.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:43.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:43.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:43.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=111, tag_tid=14, entry_tid=13803], [object_number=110, tag_tid=14, entry_tid=13802], [object_number=109, tag_tid=14, entry_tid=13801], [object_number=108, tag_tid=14, entry_tid=13800]]], state=connected] 2024-08-31T22:23:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:43.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:43.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:43.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:43.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:43.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 27 2024-08-31T22:23:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=111, tag_tid=14, entry_tid=13803], [object_number=110, tag_tid=14, entry_tid=13802], [object_number=109, tag_tid=14, entry_tid=13801], [object_number=108, tag_tid=14, entry_tid=13800]]], state=connected] 2024-08-31T22:23:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=111, tag_tid=14, entry_tid=13803' 2024-08-31T22:23:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=111, tag_tid=14, entry_tid=13803' 2024-08-31T22:23:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=111, tag_tid=14, entry_tid=13803' '!=' 'object_number=111, tag_tid=14, entry_tid=13699' 2024-08-31T22:23:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=111, tag_tid=14, entry_tid=13803' = 'object_number=111, tag_tid=14, entry_tid=13699' 2024-08-31T22:23:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=111, tag_tid=14, entry_tid=13803' 2024-08-31T22:23:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=111, tag_tid=14, entry_tid=13803' 2024-08-31T22:23:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=111, tag_tid=14, entry_tid=13803' 2024-08-31T22:23:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13803 2024-08-31T22:23:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 13803 2024-08-31T22:23:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:43.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=13842], [object_number=113, tag_tid=14, entry_tid=13841], [object_number=112, tag_tid=14, entry_tid=13840], [object_number=115, tag_tid=14, entry_tid=13839]]], state=connected] 2024-08-31T22:23:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=13842], [object_number=113, tag_tid=14, entry_tid=13841], [object_number=112, tag_tid=14, entry_tid=13840], [object_number=115, tag_tid=14, entry_tid=13839]]], state=connected] 2024-08-31T22:23:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:43.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=114, tag_tid=14, entry_tid=13842' 2024-08-31T22:23:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=114, tag_tid=14, entry_tid=13842' 2024-08-31T22:23:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=13842' '!=' 'object_number=111, tag_tid=14, entry_tid=13803' 2024-08-31T22:23:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=13842' = 'object_number=111, tag_tid=14, entry_tid=13803' 2024-08-31T22:23:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=114, tag_tid=14, entry_tid=13842' 2024-08-31T22:23:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:43.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:43.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=13842' 2024-08-31T22:23:43.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:43.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:43.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=13842' 2024-08-31T22:23:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13842 2024-08-31T22:23:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 13842 2024-08-31T22:23:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=13938], [object_number=113, tag_tid=14, entry_tid=13937], [object_number=112, tag_tid=14, entry_tid=13936], [object_number=115, tag_tid=14, entry_tid=13935]]], state=connected] 2024-08-31T22:23:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:44.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=13938], [object_number=113, tag_tid=14, entry_tid=13937], [object_number=112, tag_tid=14, entry_tid=13936], [object_number=115, tag_tid=14, entry_tid=13935]]], state=connected] 2024-08-31T22:23:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=114, tag_tid=14, entry_tid=13938' 2024-08-31T22:23:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=114, tag_tid=14, entry_tid=13938' 2024-08-31T22:23:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=13938' '!=' 'object_number=114, tag_tid=14, entry_tid=13842' 2024-08-31T22:23:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=13938' = 'object_number=114, tag_tid=14, entry_tid=13842' 2024-08-31T22:23:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=114, tag_tid=14, entry_tid=13938' 2024-08-31T22:23:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=13938' 2024-08-31T22:23:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=13938' 2024-08-31T22:23:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13938 2024-08-31T22:23:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 13938 2024-08-31T22:23:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=14022], [object_number=113, tag_tid=14, entry_tid=14021], [object_number=112, tag_tid=14, entry_tid=14020], [object_number=115, tag_tid=14, entry_tid=14019]]], state=connected] 2024-08-31T22:23:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=14022], [object_number=113, tag_tid=14, entry_tid=14021], [object_number=112, tag_tid=14, entry_tid=14020], [object_number=115, tag_tid=14, entry_tid=14019]]], state=connected] 2024-08-31T22:23:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=114, tag_tid=14, entry_tid=14022' 2024-08-31T22:23:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=114, tag_tid=14, entry_tid=14022' 2024-08-31T22:23:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=14022' '!=' 'object_number=114, tag_tid=14, entry_tid=13938' 2024-08-31T22:23:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=14022' = 'object_number=114, tag_tid=14, entry_tid=13938' 2024-08-31T22:23:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=114, tag_tid=14, entry_tid=14022' 2024-08-31T22:23:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:44.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=14022' 2024-08-31T22:23:44.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:44.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:44.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=14022' 2024-08-31T22:23:44.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:44.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:44.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14022 2024-08-31T22:23:44.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 14022 2024-08-31T22:23:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:45.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:45.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:45.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:45.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=14111], [object_number=114, tag_tid=14, entry_tid=14110], [object_number=113, tag_tid=14, entry_tid=14109], [object_number=112, tag_tid=14, entry_tid=14108]]], state=connected] 2024-08-31T22:23:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=14111], [object_number=114, tag_tid=14, entry_tid=14110], [object_number=113, tag_tid=14, entry_tid=14109], [object_number=112, tag_tid=14, entry_tid=14108]]], state=connected] 2024-08-31T22:23:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:45.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=115, tag_tid=14, entry_tid=14111' 2024-08-31T22:23:45.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=115, tag_tid=14, entry_tid=14111' 2024-08-31T22:23:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=14111' '!=' 'object_number=114, tag_tid=14, entry_tid=14022' 2024-08-31T22:23:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=14111' = 'object_number=114, tag_tid=14, entry_tid=14022' 2024-08-31T22:23:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=115, tag_tid=14, entry_tid=14111' 2024-08-31T22:23:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=14111' 2024-08-31T22:23:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=14111' 2024-08-31T22:23:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14111 2024-08-31T22:23:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 14111 2024-08-31T22:23:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:45.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=14211], [object_number=114, tag_tid=14, entry_tid=14210], [object_number=113, tag_tid=14, entry_tid=14209], [object_number=112, tag_tid=14, entry_tid=14208]]], state=connected] 2024-08-31T22:23:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=14211], [object_number=114, tag_tid=14, entry_tid=14210], [object_number=113, tag_tid=14, entry_tid=14209], [object_number=112, tag_tid=14, entry_tid=14208]]], state=connected] 2024-08-31T22:23:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=115, tag_tid=14, entry_tid=14211' 2024-08-31T22:23:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=115, tag_tid=14, entry_tid=14211' 2024-08-31T22:23:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=14211' '!=' 'object_number=115, tag_tid=14, entry_tid=14111' 2024-08-31T22:23:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=14211' = 'object_number=115, tag_tid=14, entry_tid=14111' 2024-08-31T22:23:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=115, tag_tid=14, entry_tid=14211' 2024-08-31T22:23:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=14211' 2024-08-31T22:23:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=14211' 2024-08-31T22:23:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14211 2024-08-31T22:23:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 14211 2024-08-31T22:23:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=14315], [object_number=114, tag_tid=14, entry_tid=14314], [object_number=113, tag_tid=14, entry_tid=14313], [object_number=112, tag_tid=14, entry_tid=14312]]], state=connected] 2024-08-31T22:23:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=14315], [object_number=114, tag_tid=14, entry_tid=14314], [object_number=113, tag_tid=14, entry_tid=14313], [object_number=112, tag_tid=14, entry_tid=14312]]], state=connected] 2024-08-31T22:23:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=115, tag_tid=14, entry_tid=14315' 2024-08-31T22:23:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=115, tag_tid=14, entry_tid=14315' 2024-08-31T22:23:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=14315' '!=' 'object_number=115, tag_tid=14, entry_tid=14211' 2024-08-31T22:23:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=14315' = 'object_number=115, tag_tid=14, entry_tid=14211' 2024-08-31T22:23:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=115, tag_tid=14, entry_tid=14315' 2024-08-31T22:23:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=14315' 2024-08-31T22:23:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:46.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:46.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=14315' 2024-08-31T22:23:46.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:46.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14315 2024-08-31T22:23:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 14315 2024-08-31T22:23:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:46.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:46.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:46.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:46.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:46.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=14407], [object_number=114, tag_tid=14, entry_tid=14406], [object_number=113, tag_tid=14, entry_tid=14405], [object_number=112, tag_tid=14, entry_tid=14404]]], state=connected] 2024-08-31T22:23:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=14407], [object_number=114, tag_tid=14, entry_tid=14406], [object_number=113, tag_tid=14, entry_tid=14405], [object_number=112, tag_tid=14, entry_tid=14404]]], state=connected] 2024-08-31T22:23:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=115, tag_tid=14, entry_tid=14407' 2024-08-31T22:23:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=115, tag_tid=14, entry_tid=14407' 2024-08-31T22:23:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=14407' '!=' 'object_number=115, tag_tid=14, entry_tid=14315' 2024-08-31T22:23:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=14407' = 'object_number=115, tag_tid=14, entry_tid=14315' 2024-08-31T22:23:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=115, tag_tid=14, entry_tid=14407' 2024-08-31T22:23:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=14407' 2024-08-31T22:23:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:46.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:46.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:46.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:46.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=14407' 2024-08-31T22:23:46.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:46.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14407 2024-08-31T22:23:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 14407 2024-08-31T22:23:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=14502], [object_number=113, tag_tid=14, entry_tid=14501], [object_number=112, tag_tid=14, entry_tid=14500], [object_number=115, tag_tid=14, entry_tid=14499]]], state=connected] 2024-08-31T22:23:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=14502], [object_number=113, tag_tid=14, entry_tid=14501], [object_number=112, tag_tid=14, entry_tid=14500], [object_number=115, tag_tid=14, entry_tid=14499]]], state=connected] 2024-08-31T22:23:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=114, tag_tid=14, entry_tid=14502' 2024-08-31T22:23:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=114, tag_tid=14, entry_tid=14502' 2024-08-31T22:23:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=14502' '!=' 'object_number=115, tag_tid=14, entry_tid=14407' 2024-08-31T22:23:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=14502' = 'object_number=115, tag_tid=14, entry_tid=14407' 2024-08-31T22:23:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=114, tag_tid=14, entry_tid=14502' 2024-08-31T22:23:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=14502' 2024-08-31T22:23:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=14502' 2024-08-31T22:23:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14502 2024-08-31T22:23:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 14502 2024-08-31T22:23:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=112, tag_tid=14, entry_tid=14588], [object_number=115, tag_tid=14, entry_tid=14587], [object_number=114, tag_tid=14, entry_tid=14586], [object_number=113, tag_tid=14, entry_tid=14585]]], state=connected] 2024-08-31T22:23:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:47.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=112, tag_tid=14, entry_tid=14588], [object_number=115, tag_tid=14, entry_tid=14587], [object_number=114, tag_tid=14, entry_tid=14586], [object_number=113, tag_tid=14, entry_tid=14585]]], state=connected] 2024-08-31T22:23:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=112, tag_tid=14, entry_tid=14588' 2024-08-31T22:23:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=112, tag_tid=14, entry_tid=14588' 2024-08-31T22:23:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=112, tag_tid=14, entry_tid=14588' '!=' 'object_number=114, tag_tid=14, entry_tid=14502' 2024-08-31T22:23:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=112, tag_tid=14, entry_tid=14588' = 'object_number=114, tag_tid=14, entry_tid=14502' 2024-08-31T22:23:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=112, tag_tid=14, entry_tid=14588' 2024-08-31T22:23:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=112, tag_tid=14, entry_tid=14588' 2024-08-31T22:23:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=112, tag_tid=14, entry_tid=14588' 2024-08-31T22:23:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14588 2024-08-31T22:23:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 14588 2024-08-31T22:23:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=14686], [object_number=113, tag_tid=14, entry_tid=14685], [object_number=112, tag_tid=14, entry_tid=14684], [object_number=115, tag_tid=14, entry_tid=14683]]], state=connected] 2024-08-31T22:23:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=14686], [object_number=113, tag_tid=14, entry_tid=14685], [object_number=112, tag_tid=14, entry_tid=14684], [object_number=115, tag_tid=14, entry_tid=14683]]], state=connected] 2024-08-31T22:23:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=114, tag_tid=14, entry_tid=14686' 2024-08-31T22:23:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=114, tag_tid=14, entry_tid=14686' 2024-08-31T22:23:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=14686' '!=' 'object_number=112, tag_tid=14, entry_tid=14588' 2024-08-31T22:23:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=14686' = 'object_number=112, tag_tid=14, entry_tid=14588' 2024-08-31T22:23:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=114, tag_tid=14, entry_tid=14686' 2024-08-31T22:23:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=14686' 2024-08-31T22:23:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=14686' 2024-08-31T22:23:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14686 2024-08-31T22:23:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 14686 2024-08-31T22:23:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=14795], [object_number=114, tag_tid=14, entry_tid=14794], [object_number=113, tag_tid=14, entry_tid=14793], [object_number=112, tag_tid=14, entry_tid=14792]]], state=connected] 2024-08-31T22:23:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=14795], [object_number=114, tag_tid=14, entry_tid=14794], [object_number=113, tag_tid=14, entry_tid=14793], [object_number=112, tag_tid=14, entry_tid=14792]]], state=connected] 2024-08-31T22:23:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=115, tag_tid=14, entry_tid=14795' 2024-08-31T22:23:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=115, tag_tid=14, entry_tid=14795' 2024-08-31T22:23:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=14795' '!=' 'object_number=114, tag_tid=14, entry_tid=14686' 2024-08-31T22:23:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=14795' = 'object_number=114, tag_tid=14, entry_tid=14686' 2024-08-31T22:23:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=115, tag_tid=14, entry_tid=14795' 2024-08-31T22:23:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=14795' 2024-08-31T22:23:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=14795' 2024-08-31T22:23:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14795 2024-08-31T22:23:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 14795 2024-08-31T22:23:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=14891], [object_number=114, tag_tid=14, entry_tid=14890], [object_number=113, tag_tid=14, entry_tid=14889], [object_number=112, tag_tid=14, entry_tid=14888]]], state=connected] 2024-08-31T22:23:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=14891], [object_number=114, tag_tid=14, entry_tid=14890], [object_number=113, tag_tid=14, entry_tid=14889], [object_number=112, tag_tid=14, entry_tid=14888]]], state=connected] 2024-08-31T22:23:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=115, tag_tid=14, entry_tid=14891' 2024-08-31T22:23:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=115, tag_tid=14, entry_tid=14891' 2024-08-31T22:23:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=14891' '!=' 'object_number=115, tag_tid=14, entry_tid=14795' 2024-08-31T22:23:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=14891' = 'object_number=115, tag_tid=14, entry_tid=14795' 2024-08-31T22:23:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=115, tag_tid=14, entry_tid=14891' 2024-08-31T22:23:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=14891' 2024-08-31T22:23:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=14891' 2024-08-31T22:23:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14891 2024-08-31T22:23:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 14891 2024-08-31T22:23:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:50.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=15066], [object_number=113, tag_tid=14, entry_tid=15065], [object_number=112, tag_tid=14, entry_tid=15064], [object_number=115, tag_tid=14, entry_tid=15063]]], state=connected] 2024-08-31T22:23:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=15066], [object_number=113, tag_tid=14, entry_tid=15065], [object_number=112, tag_tid=14, entry_tid=15064], [object_number=115, tag_tid=14, entry_tid=15063]]], state=connected] 2024-08-31T22:23:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=114, tag_tid=14, entry_tid=15066' 2024-08-31T22:23:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=114, tag_tid=14, entry_tid=15066' 2024-08-31T22:23:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=15066' '!=' 'object_number=115, tag_tid=14, entry_tid=14891' 2024-08-31T22:23:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=15066' = 'object_number=115, tag_tid=14, entry_tid=14891' 2024-08-31T22:23:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=114, tag_tid=14, entry_tid=15066' 2024-08-31T22:23:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=15066' 2024-08-31T22:23:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=15066' 2024-08-31T22:23:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15066 2024-08-31T22:23:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 15066 2024-08-31T22:23:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:50.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:50.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:50.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=112, tag_tid=14, entry_tid=15152], [object_number=115, tag_tid=14, entry_tid=15151], [object_number=114, tag_tid=14, entry_tid=15150], [object_number=113, tag_tid=14, entry_tid=15149]]], state=connected] 2024-08-31T22:23:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=112, tag_tid=14, entry_tid=15152], [object_number=115, tag_tid=14, entry_tid=15151], [object_number=114, tag_tid=14, entry_tid=15150], [object_number=113, tag_tid=14, entry_tid=15149]]], state=connected] 2024-08-31T22:23:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=112, tag_tid=14, entry_tid=15152' 2024-08-31T22:23:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=112, tag_tid=14, entry_tid=15152' 2024-08-31T22:23:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=112, tag_tid=14, entry_tid=15152' '!=' 'object_number=114, tag_tid=14, entry_tid=15066' 2024-08-31T22:23:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=112, tag_tid=14, entry_tid=15152' = 'object_number=114, tag_tid=14, entry_tid=15066' 2024-08-31T22:23:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=112, tag_tid=14, entry_tid=15152' 2024-08-31T22:23:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=112, tag_tid=14, entry_tid=15152' 2024-08-31T22:23:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=112, tag_tid=14, entry_tid=15152' 2024-08-31T22:23:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15152 2024-08-31T22:23:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 15152 2024-08-31T22:23:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:51.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:51.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:51.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:51.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:51.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:51.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:51.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=15243], [object_number=114, tag_tid=14, entry_tid=15242], [object_number=113, tag_tid=14, entry_tid=15241], [object_number=112, tag_tid=14, entry_tid=15240]]], state=connected] 2024-08-31T22:23:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:51.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:51.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:51.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=15243], [object_number=114, tag_tid=14, entry_tid=15242], [object_number=113, tag_tid=14, entry_tid=15241], [object_number=112, tag_tid=14, entry_tid=15240]]], state=connected] 2024-08-31T22:23:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=115, tag_tid=14, entry_tid=15243' 2024-08-31T22:23:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=115, tag_tid=14, entry_tid=15243' 2024-08-31T22:23:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=15243' '!=' 'object_number=112, tag_tid=14, entry_tid=15152' 2024-08-31T22:23:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=15243' = 'object_number=112, tag_tid=14, entry_tid=15152' 2024-08-31T22:23:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=115, tag_tid=14, entry_tid=15243' 2024-08-31T22:23:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=15243' 2024-08-31T22:23:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=15243' 2024-08-31T22:23:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15243 2024-08-31T22:23:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 15243 2024-08-31T22:23:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=15334], [object_number=113, tag_tid=14, entry_tid=15333], [object_number=112, tag_tid=14, entry_tid=15332], [object_number=115, tag_tid=14, entry_tid=15331]]], state=connected] 2024-08-31T22:23:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=15334], [object_number=113, tag_tid=14, entry_tid=15333], [object_number=112, tag_tid=14, entry_tid=15332], [object_number=115, tag_tid=14, entry_tid=15331]]], state=connected] 2024-08-31T22:23:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=114, tag_tid=14, entry_tid=15334' 2024-08-31T22:23:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=114, tag_tid=14, entry_tid=15334' 2024-08-31T22:23:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=15334' '!=' 'object_number=115, tag_tid=14, entry_tid=15243' 2024-08-31T22:23:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=15334' = 'object_number=115, tag_tid=14, entry_tid=15243' 2024-08-31T22:23:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=114, tag_tid=14, entry_tid=15334' 2024-08-31T22:23:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:51.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:51.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=15334' 2024-08-31T22:23:51.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=15334' 2024-08-31T22:23:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15334 2024-08-31T22:23:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 15334 2024-08-31T22:23:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:52.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:52.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:52.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:52.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=113, tag_tid=14, entry_tid=15421], [object_number=112, tag_tid=14, entry_tid=15420], [object_number=115, tag_tid=14, entry_tid=15419], [object_number=114, tag_tid=14, entry_tid=15418]]], state=connected] 2024-08-31T22:23:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=113, tag_tid=14, entry_tid=15421], [object_number=112, tag_tid=14, entry_tid=15420], [object_number=115, tag_tid=14, entry_tid=15419], [object_number=114, tag_tid=14, entry_tid=15418]]], state=connected] 2024-08-31T22:23:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=113, tag_tid=14, entry_tid=15421' 2024-08-31T22:23:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=113, tag_tid=14, entry_tid=15421' 2024-08-31T22:23:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=113, tag_tid=14, entry_tid=15421' '!=' 'object_number=114, tag_tid=14, entry_tid=15334' 2024-08-31T22:23:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=113, tag_tid=14, entry_tid=15421' = 'object_number=114, tag_tid=14, entry_tid=15334' 2024-08-31T22:23:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=113, tag_tid=14, entry_tid=15421' 2024-08-31T22:23:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=113, tag_tid=14, entry_tid=15421' 2024-08-31T22:23:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=113, tag_tid=14, entry_tid=15421' 2024-08-31T22:23:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15421 2024-08-31T22:23:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 15421 2024-08-31T22:23:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=15514], [object_number=113, tag_tid=14, entry_tid=15513], [object_number=112, tag_tid=14, entry_tid=15512], [object_number=115, tag_tid=14, entry_tid=15511]]], state=connected] 2024-08-31T22:23:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=15514], [object_number=113, tag_tid=14, entry_tid=15513], [object_number=112, tag_tid=14, entry_tid=15512], [object_number=115, tag_tid=14, entry_tid=15511]]], state=connected] 2024-08-31T22:23:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:52.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=114, tag_tid=14, entry_tid=15514' 2024-08-31T22:23:52.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=114, tag_tid=14, entry_tid=15514' 2024-08-31T22:23:52.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=15514' '!=' 'object_number=113, tag_tid=14, entry_tid=15421' 2024-08-31T22:23:52.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:52.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=15514' = 'object_number=113, tag_tid=14, entry_tid=15421' 2024-08-31T22:23:52.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=114, tag_tid=14, entry_tid=15514' 2024-08-31T22:23:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:52.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=15514' 2024-08-31T22:23:52.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:52.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:52.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:52.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=15514' 2024-08-31T22:23:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15514 2024-08-31T22:23:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 15514 2024-08-31T22:23:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:52.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=15622], [object_number=113, tag_tid=14, entry_tid=15621], [object_number=112, tag_tid=14, entry_tid=15620], [object_number=115, tag_tid=14, entry_tid=15619]]], state=connected] 2024-08-31T22:23:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=15622], [object_number=113, tag_tid=14, entry_tid=15621], [object_number=112, tag_tid=14, entry_tid=15620], [object_number=115, tag_tid=14, entry_tid=15619]]], state=connected] 2024-08-31T22:23:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=114, tag_tid=14, entry_tid=15622' 2024-08-31T22:23:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=114, tag_tid=14, entry_tid=15622' 2024-08-31T22:23:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=15622' '!=' 'object_number=114, tag_tid=14, entry_tid=15514' 2024-08-31T22:23:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=15622' = 'object_number=114, tag_tid=14, entry_tid=15514' 2024-08-31T22:23:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=114, tag_tid=14, entry_tid=15622' 2024-08-31T22:23:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=15622' 2024-08-31T22:23:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:53.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:53.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=15622' 2024-08-31T22:23:53.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:53.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15622 2024-08-31T22:23:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 15622 2024-08-31T22:23:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=15718], [object_number=113, tag_tid=14, entry_tid=15717], [object_number=112, tag_tid=14, entry_tid=15716], [object_number=115, tag_tid=14, entry_tid=15715]]], state=connected] 2024-08-31T22:23:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=15718], [object_number=113, tag_tid=14, entry_tid=15717], [object_number=112, tag_tid=14, entry_tid=15716], [object_number=115, tag_tid=14, entry_tid=15715]]], state=connected] 2024-08-31T22:23:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=114, tag_tid=14, entry_tid=15718' 2024-08-31T22:23:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=114, tag_tid=14, entry_tid=15718' 2024-08-31T22:23:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=15718' '!=' 'object_number=114, tag_tid=14, entry_tid=15622' 2024-08-31T22:23:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=15718' = 'object_number=114, tag_tid=14, entry_tid=15622' 2024-08-31T22:23:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=114, tag_tid=14, entry_tid=15718' 2024-08-31T22:23:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:53.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=15718' 2024-08-31T22:23:53.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:53.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=15718' 2024-08-31T22:23:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15718 2024-08-31T22:23:53.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 15718 2024-08-31T22:23:53.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:53.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:53.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:54.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=15810], [object_number=113, tag_tid=14, entry_tid=15809], [object_number=112, tag_tid=14, entry_tid=15808], [object_number=115, tag_tid=14, entry_tid=15807]]], state=connected] 2024-08-31T22:23:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=15810], [object_number=113, tag_tid=14, entry_tid=15809], [object_number=112, tag_tid=14, entry_tid=15808], [object_number=115, tag_tid=14, entry_tid=15807]]], state=connected] 2024-08-31T22:23:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=114, tag_tid=14, entry_tid=15810' 2024-08-31T22:23:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=114, tag_tid=14, entry_tid=15810' 2024-08-31T22:23:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=15810' '!=' 'object_number=114, tag_tid=14, entry_tid=15718' 2024-08-31T22:23:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=15810' = 'object_number=114, tag_tid=14, entry_tid=15718' 2024-08-31T22:23:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=114, tag_tid=14, entry_tid=15810' 2024-08-31T22:23:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=15810' 2024-08-31T22:23:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=15810' 2024-08-31T22:23:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15810 2024-08-31T22:23:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 15810 2024-08-31T22:23:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:54.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=15903], [object_number=114, tag_tid=14, entry_tid=15902], [object_number=113, tag_tid=14, entry_tid=15901], [object_number=112, tag_tid=14, entry_tid=15900]]], state=connected] 2024-08-31T22:23:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=15903], [object_number=114, tag_tid=14, entry_tid=15902], [object_number=113, tag_tid=14, entry_tid=15901], [object_number=112, tag_tid=14, entry_tid=15900]]], state=connected] 2024-08-31T22:23:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=115, tag_tid=14, entry_tid=15903' 2024-08-31T22:23:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=115, tag_tid=14, entry_tid=15903' 2024-08-31T22:23:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=15903' '!=' 'object_number=114, tag_tid=14, entry_tid=15810' 2024-08-31T22:23:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=15903' = 'object_number=114, tag_tid=14, entry_tid=15810' 2024-08-31T22:23:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=115, tag_tid=14, entry_tid=15903' 2024-08-31T22:23:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=15903' 2024-08-31T22:23:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=15903' 2024-08-31T22:23:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:54.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15903 2024-08-31T22:23:54.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 15903 2024-08-31T22:23:54.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:54.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=15995], [object_number=114, tag_tid=14, entry_tid=15994], [object_number=113, tag_tid=14, entry_tid=15993], [object_number=112, tag_tid=14, entry_tid=15992]]], state=connected] 2024-08-31T22:23:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:55.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=15995], [object_number=114, tag_tid=14, entry_tid=15994], [object_number=113, tag_tid=14, entry_tid=15993], [object_number=112, tag_tid=14, entry_tid=15992]]], state=connected] 2024-08-31T22:23:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=115, tag_tid=14, entry_tid=15995' 2024-08-31T22:23:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=115, tag_tid=14, entry_tid=15995' 2024-08-31T22:23:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=15995' '!=' 'object_number=115, tag_tid=14, entry_tid=15903' 2024-08-31T22:23:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=15995' = 'object_number=115, tag_tid=14, entry_tid=15903' 2024-08-31T22:23:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=115, tag_tid=14, entry_tid=15995' 2024-08-31T22:23:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=15995' 2024-08-31T22:23:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:55.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=15995' 2024-08-31T22:23:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15995 2024-08-31T22:23:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 15995 2024-08-31T22:23:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=113, tag_tid=14, entry_tid=16081], [object_number=112, tag_tid=14, entry_tid=16080], [object_number=115, tag_tid=14, entry_tid=16079], [object_number=114, tag_tid=14, entry_tid=16078]]], state=connected] 2024-08-31T22:23:55.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=113, tag_tid=14, entry_tid=16081], [object_number=112, tag_tid=14, entry_tid=16080], [object_number=115, tag_tid=14, entry_tid=16079], [object_number=114, tag_tid=14, entry_tid=16078]]], state=connected] 2024-08-31T22:23:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=113, tag_tid=14, entry_tid=16081' 2024-08-31T22:23:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=113, tag_tid=14, entry_tid=16081' 2024-08-31T22:23:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=113, tag_tid=14, entry_tid=16081' '!=' 'object_number=115, tag_tid=14, entry_tid=15995' 2024-08-31T22:23:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=113, tag_tid=14, entry_tid=16081' = 'object_number=115, tag_tid=14, entry_tid=15995' 2024-08-31T22:23:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=113, tag_tid=14, entry_tid=16081' 2024-08-31T22:23:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=113, tag_tid=14, entry_tid=16081' 2024-08-31T22:23:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=113, tag_tid=14, entry_tid=16081' 2024-08-31T22:23:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16081 2024-08-31T22:23:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 16081 2024-08-31T22:23:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:56.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:56.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=113, tag_tid=14, entry_tid=16165], [object_number=112, tag_tid=14, entry_tid=16164], [object_number=115, tag_tid=14, entry_tid=16163], [object_number=114, tag_tid=14, entry_tid=16162]]], state=connected] 2024-08-31T22:23:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=113, tag_tid=14, entry_tid=16165], [object_number=112, tag_tid=14, entry_tid=16164], [object_number=115, tag_tid=14, entry_tid=16163], [object_number=114, tag_tid=14, entry_tid=16162]]], state=connected] 2024-08-31T22:23:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=113, tag_tid=14, entry_tid=16165' 2024-08-31T22:23:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=113, tag_tid=14, entry_tid=16165' 2024-08-31T22:23:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=113, tag_tid=14, entry_tid=16165' '!=' 'object_number=113, tag_tid=14, entry_tid=16081' 2024-08-31T22:23:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=113, tag_tid=14, entry_tid=16165' = 'object_number=113, tag_tid=14, entry_tid=16081' 2024-08-31T22:23:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=113, tag_tid=14, entry_tid=16165' 2024-08-31T22:23:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=113, tag_tid=14, entry_tid=16165' 2024-08-31T22:23:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=113, tag_tid=14, entry_tid=16165' 2024-08-31T22:23:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16165 2024-08-31T22:23:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 16165 2024-08-31T22:23:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:56.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:56.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=113, tag_tid=14, entry_tid=16257], [object_number=112, tag_tid=14, entry_tid=16256], [object_number=115, tag_tid=14, entry_tid=16255], [object_number=114, tag_tid=14, entry_tid=16254]]], state=connected] 2024-08-31T22:23:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:56.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:56.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:56.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:56.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:56.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:56.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=113, tag_tid=14, entry_tid=16257], [object_number=112, tag_tid=14, entry_tid=16256], [object_number=115, tag_tid=14, entry_tid=16255], [object_number=114, tag_tid=14, entry_tid=16254]]], state=connected] 2024-08-31T22:23:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=113, tag_tid=14, entry_tid=16257' 2024-08-31T22:23:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=113, tag_tid=14, entry_tid=16257' 2024-08-31T22:23:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=113, tag_tid=14, entry_tid=16257' '!=' 'object_number=113, tag_tid=14, entry_tid=16165' 2024-08-31T22:23:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=113, tag_tid=14, entry_tid=16257' = 'object_number=113, tag_tid=14, entry_tid=16165' 2024-08-31T22:23:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=113, tag_tid=14, entry_tid=16257' 2024-08-31T22:23:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:56.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:56.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=113, tag_tid=14, entry_tid=16257' 2024-08-31T22:23:56.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:56.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=113, tag_tid=14, entry_tid=16257' 2024-08-31T22:23:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16257 2024-08-31T22:23:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 16257 2024-08-31T22:23:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=113, tag_tid=14, entry_tid=16345], [object_number=112, tag_tid=14, entry_tid=16344], [object_number=115, tag_tid=14, entry_tid=16343], [object_number=114, tag_tid=14, entry_tid=16342]]], state=connected] 2024-08-31T22:23:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=113, tag_tid=14, entry_tid=16345], [object_number=112, tag_tid=14, entry_tid=16344], [object_number=115, tag_tid=14, entry_tid=16343], [object_number=114, tag_tid=14, entry_tid=16342]]], state=connected] 2024-08-31T22:23:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:57.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=113, tag_tid=14, entry_tid=16345' 2024-08-31T22:23:57.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=113, tag_tid=14, entry_tid=16345' 2024-08-31T22:23:57.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=113, tag_tid=14, entry_tid=16345' '!=' 'object_number=113, tag_tid=14, entry_tid=16257' 2024-08-31T22:23:57.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:57.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=113, tag_tid=14, entry_tid=16345' = 'object_number=113, tag_tid=14, entry_tid=16257' 2024-08-31T22:23:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=113, tag_tid=14, entry_tid=16345' 2024-08-31T22:23:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=113, tag_tid=14, entry_tid=16345' 2024-08-31T22:23:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=113, tag_tid=14, entry_tid=16345' 2024-08-31T22:23:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16345 2024-08-31T22:23:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 16345 2024-08-31T22:23:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:57.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:57.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:57.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:57.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:57.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:57.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:57.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=16426], [object_number=113, tag_tid=14, entry_tid=16425], [object_number=112, tag_tid=14, entry_tid=16424], [object_number=115, tag_tid=14, entry_tid=16423]]], state=connected] 2024-08-31T22:23:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:57.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:57.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:57.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=16426], [object_number=113, tag_tid=14, entry_tid=16425], [object_number=112, tag_tid=14, entry_tid=16424], [object_number=115, tag_tid=14, entry_tid=16423]]], state=connected] 2024-08-31T22:23:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=114, tag_tid=14, entry_tid=16426' 2024-08-31T22:23:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=114, tag_tid=14, entry_tid=16426' 2024-08-31T22:23:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=16426' '!=' 'object_number=113, tag_tid=14, entry_tid=16345' 2024-08-31T22:23:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=16426' = 'object_number=113, tag_tid=14, entry_tid=16345' 2024-08-31T22:23:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=114, tag_tid=14, entry_tid=16426' 2024-08-31T22:23:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=16426' 2024-08-31T22:23:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=16426' 2024-08-31T22:23:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16426 2024-08-31T22:23:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 16426 2024-08-31T22:23:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:58.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=16506], [object_number=113, tag_tid=14, entry_tid=16505], [object_number=112, tag_tid=14, entry_tid=16504], [object_number=115, tag_tid=14, entry_tid=16503]]], state=connected] 2024-08-31T22:23:58.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=114, tag_tid=14, entry_tid=16506], [object_number=113, tag_tid=14, entry_tid=16505], [object_number=112, tag_tid=14, entry_tid=16504], [object_number=115, tag_tid=14, entry_tid=16503]]], state=connected] 2024-08-31T22:23:58.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=114, tag_tid=14, entry_tid=16506' 2024-08-31T22:23:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=114, tag_tid=14, entry_tid=16506' 2024-08-31T22:23:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=16506' '!=' 'object_number=114, tag_tid=14, entry_tid=16426' 2024-08-31T22:23:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=114, tag_tid=14, entry_tid=16506' = 'object_number=114, tag_tid=14, entry_tid=16426' 2024-08-31T22:23:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=114, tag_tid=14, entry_tid=16506' 2024-08-31T22:23:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=16506' 2024-08-31T22:23:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:58.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=114, tag_tid=14, entry_tid=16506' 2024-08-31T22:23:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16506 2024-08-31T22:23:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 16506 2024-08-31T22:23:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:58.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:58.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:58.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:58.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:58.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=113, tag_tid=14, entry_tid=16597], [object_number=112, tag_tid=14, entry_tid=16596], [object_number=115, tag_tid=14, entry_tid=16595], [object_number=114, tag_tid=14, entry_tid=16594]]], state=connected] 2024-08-31T22:23:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=113, tag_tid=14, entry_tid=16597], [object_number=112, tag_tid=14, entry_tid=16596], [object_number=115, tag_tid=14, entry_tid=16595], [object_number=114, tag_tid=14, entry_tid=16594]]], state=connected] 2024-08-31T22:23:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=113, tag_tid=14, entry_tid=16597' 2024-08-31T22:23:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=113, tag_tid=14, entry_tid=16597' 2024-08-31T22:23:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=113, tag_tid=14, entry_tid=16597' '!=' 'object_number=114, tag_tid=14, entry_tid=16506' 2024-08-31T22:23:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=113, tag_tid=14, entry_tid=16597' = 'object_number=114, tag_tid=14, entry_tid=16506' 2024-08-31T22:23:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=113, tag_tid=14, entry_tid=16597' 2024-08-31T22:23:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=113, tag_tid=14, entry_tid=16597' 2024-08-31T22:23:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=113, tag_tid=14, entry_tid=16597' 2024-08-31T22:23:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16597 2024-08-31T22:23:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 16597 2024-08-31T22:23:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=112, tag_tid=14, entry_tid=16684], [object_number=115, tag_tid=14, entry_tid=16683], [object_number=114, tag_tid=14, entry_tid=16682], [object_number=113, tag_tid=14, entry_tid=16681]]], state=connected] 2024-08-31T22:23:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=112, tag_tid=14, entry_tid=16684], [object_number=115, tag_tid=14, entry_tid=16683], [object_number=114, tag_tid=14, entry_tid=16682], [object_number=113, tag_tid=14, entry_tid=16681]]], state=connected] 2024-08-31T22:23:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=112, tag_tid=14, entry_tid=16684' 2024-08-31T22:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=112, tag_tid=14, entry_tid=16684' 2024-08-31T22:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=112, tag_tid=14, entry_tid=16684' '!=' 'object_number=113, tag_tid=14, entry_tid=16597' 2024-08-31T22:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=112, tag_tid=14, entry_tid=16684' = 'object_number=113, tag_tid=14, entry_tid=16597' 2024-08-31T22:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=112, tag_tid=14, entry_tid=16684' 2024-08-31T22:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=112, tag_tid=14, entry_tid=16684' 2024-08-31T22:23:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=112, tag_tid=14, entry_tid=16684' 2024-08-31T22:23:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16684 2024-08-31T22:23:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 16684 2024-08-31T22:23:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:23:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:23:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:23:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:23:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:23:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:23:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:23:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:23:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:23:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:23:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:23:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:23:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:23:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:23:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:23:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:23:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:23:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:23:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:23:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:23:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:23:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:23:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:23:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:23:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:23:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:23:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:23:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:23:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:23:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:23:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:23:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:23:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:23:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:23:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:23:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:23:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:23:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:23:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:23:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:23:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:23:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:23:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:23:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:23:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:23:59.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:23:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:23:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:23:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:23:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:23:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:23:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=112, tag_tid=14, entry_tid=16772], [object_number=115, tag_tid=14, entry_tid=16771], [object_number=114, tag_tid=14, entry_tid=16770], [object_number=113, tag_tid=14, entry_tid=16769]]], state=connected] 2024-08-31T22:23:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:23:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:23:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:23:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:23:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:23:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:23:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:23:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:23:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:23:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:23:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:23:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:23:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:23:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:23:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=112, tag_tid=14, entry_tid=16772], [object_number=115, tag_tid=14, entry_tid=16771], [object_number=114, tag_tid=14, entry_tid=16770], [object_number=113, tag_tid=14, entry_tid=16769]]], state=connected] 2024-08-31T22:23:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:23:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=112, tag_tid=14, entry_tid=16772' 2024-08-31T22:23:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=112, tag_tid=14, entry_tid=16772' 2024-08-31T22:23:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=112, tag_tid=14, entry_tid=16772' '!=' 'object_number=112, tag_tid=14, entry_tid=16684' 2024-08-31T22:23:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:23:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=112, tag_tid=14, entry_tid=16772' = 'object_number=112, tag_tid=14, entry_tid=16684' 2024-08-31T22:23:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=112, tag_tid=14, entry_tid=16772' 2024-08-31T22:23:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:23:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=112, tag_tid=14, entry_tid=16772' 2024-08-31T22:23:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:23:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:23:59.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:23:59.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:59.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:59.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:23:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=112, tag_tid=14, entry_tid=16772' 2024-08-31T22:23:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:23:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:23:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16772 2024-08-31T22:23:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 16772 2024-08-31T22:23:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:23:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:23:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:00.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:00.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:00.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:00.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:00.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:00.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:00.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:00.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:00.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:00.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:00.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:00.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:00.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:00.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:00.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:00.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:00.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:00.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:00.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:00.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:00.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:00.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:00.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:00.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:00.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:00.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:24:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=16855], [object_number=114, tag_tid=14, entry_tid=16854], [object_number=113, tag_tid=14, entry_tid=16853], [object_number=112, tag_tid=14, entry_tid=16852]]], state=connected] 2024-08-31T22:24:00.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:00.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:00.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:00.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:24:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=115, tag_tid=14, entry_tid=16855], [object_number=114, tag_tid=14, entry_tid=16854], [object_number=113, tag_tid=14, entry_tid=16853], [object_number=112, tag_tid=14, entry_tid=16852]]], state=connected] 2024-08-31T22:24:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:00.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=115, tag_tid=14, entry_tid=16855' 2024-08-31T22:24:00.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=115, tag_tid=14, entry_tid=16855' 2024-08-31T22:24:00.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=16855' '!=' 'object_number=112, tag_tid=14, entry_tid=16772' 2024-08-31T22:24:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=115, tag_tid=14, entry_tid=16855' = 'object_number=112, tag_tid=14, entry_tid=16772' 2024-08-31T22:24:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=115, tag_tid=14, entry_tid=16855' 2024-08-31T22:24:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:00.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=16855' 2024-08-31T22:24:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=115, tag_tid=14, entry_tid=16855' 2024-08-31T22:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16855 2024-08-31T22:24:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 16855 2024-08-31T22:24:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:00.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:00.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:00.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:00.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:00.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:00.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:24:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=113, tag_tid=14, entry_tid=16945], [object_number=112, tag_tid=14, entry_tid=16944], [object_number=115, tag_tid=14, entry_tid=16943], [object_number=114, tag_tid=14, entry_tid=16942]]], state=connected] 2024-08-31T22:24:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:24:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=113, tag_tid=14, entry_tid=16945], [object_number=112, tag_tid=14, entry_tid=16944], [object_number=115, tag_tid=14, entry_tid=16943], [object_number=114, tag_tid=14, entry_tid=16942]]], state=connected] 2024-08-31T22:24:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=113, tag_tid=14, entry_tid=16945' 2024-08-31T22:24:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=113, tag_tid=14, entry_tid=16945' 2024-08-31T22:24:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=113, tag_tid=14, entry_tid=16945' '!=' 'object_number=115, tag_tid=14, entry_tid=16855' 2024-08-31T22:24:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=113, tag_tid=14, entry_tid=16945' = 'object_number=115, tag_tid=14, entry_tid=16855' 2024-08-31T22:24:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=113, tag_tid=14, entry_tid=16945' 2024-08-31T22:24:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=113, tag_tid=14, entry_tid=16945' 2024-08-31T22:24:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:00.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=113, tag_tid=14, entry_tid=16945' 2024-08-31T22:24:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16945 2024-08-31T22:24:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 16945 2024-08-31T22:24:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:24:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=112, tag_tid=14, entry_tid=17036], [object_number=115, tag_tid=14, entry_tid=17035], [object_number=114, tag_tid=14, entry_tid=17034], [object_number=113, tag_tid=14, entry_tid=17033]]], state=connected] 2024-08-31T22:24:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:01.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:01.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:01.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:24:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=112, tag_tid=14, entry_tid=17036], [object_number=115, tag_tid=14, entry_tid=17035], [object_number=114, tag_tid=14, entry_tid=17034], [object_number=113, tag_tid=14, entry_tid=17033]]], state=connected] 2024-08-31T22:24:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=112, tag_tid=14, entry_tid=17036' 2024-08-31T22:24:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=112, tag_tid=14, entry_tid=17036' 2024-08-31T22:24:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=112, tag_tid=14, entry_tid=17036' '!=' 'object_number=113, tag_tid=14, entry_tid=16945' 2024-08-31T22:24:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=112, tag_tid=14, entry_tid=17036' = 'object_number=113, tag_tid=14, entry_tid=16945' 2024-08-31T22:24:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=112, tag_tid=14, entry_tid=17036' 2024-08-31T22:24:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=112, tag_tid=14, entry_tid=17036' 2024-08-31T22:24:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=112, tag_tid=14, entry_tid=17036' 2024-08-31T22:24:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17036 2024-08-31T22:24:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 17036 2024-08-31T22:24:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:01.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:01.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:01.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:01.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:24:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=113, tag_tid=14, entry_tid=17129], [object_number=112, tag_tid=14, entry_tid=17128], [object_number=115, tag_tid=14, entry_tid=17127], [object_number=114, tag_tid=14, entry_tid=17126]]], state=connected] 2024-08-31T22:24:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:24:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=113, tag_tid=14, entry_tid=17129], [object_number=112, tag_tid=14, entry_tid=17128], [object_number=115, tag_tid=14, entry_tid=17127], [object_number=114, tag_tid=14, entry_tid=17126]]], state=connected] 2024-08-31T22:24:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=113, tag_tid=14, entry_tid=17129' 2024-08-31T22:24:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=113, tag_tid=14, entry_tid=17129' 2024-08-31T22:24:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=113, tag_tid=14, entry_tid=17129' '!=' 'object_number=112, tag_tid=14, entry_tid=17036' 2024-08-31T22:24:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=113, tag_tid=14, entry_tid=17129' = 'object_number=112, tag_tid=14, entry_tid=17036' 2024-08-31T22:24:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=113, tag_tid=14, entry_tid=17129' 2024-08-31T22:24:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=113, tag_tid=14, entry_tid=17129' 2024-08-31T22:24:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=113, tag_tid=14, entry_tid=17129' 2024-08-31T22:24:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17129 2024-08-31T22:24:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 17129 2024-08-31T22:24:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:02.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:24:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=117, tag_tid=14, entry_tid=17181], [object_number=112, tag_tid=14, entry_tid=17180], [object_number=115, tag_tid=14, entry_tid=17179], [object_number=118, tag_tid=14, entry_tid=17178]]], state=connected] 2024-08-31T22:24:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:02.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:02.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 28 2024-08-31T22:24:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=117, tag_tid=14, entry_tid=17181], [object_number=112, tag_tid=14, entry_tid=17180], [object_number=115, tag_tid=14, entry_tid=17179], [object_number=118, tag_tid=14, entry_tid=17178]]], state=connected] 2024-08-31T22:24:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=117, tag_tid=14, entry_tid=17181' 2024-08-31T22:24:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=117, tag_tid=14, entry_tid=17181' 2024-08-31T22:24:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=117, tag_tid=14, entry_tid=17181' '!=' 'object_number=113, tag_tid=14, entry_tid=17129' 2024-08-31T22:24:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=117, tag_tid=14, entry_tid=17181' = 'object_number=113, tag_tid=14, entry_tid=17129' 2024-08-31T22:24:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=117, tag_tid=14, entry_tid=17181' 2024-08-31T22:24:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=117, tag_tid=14, entry_tid=17181' 2024-08-31T22:24:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=117, tag_tid=14, entry_tid=17181' 2024-08-31T22:24:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17181 2024-08-31T22:24:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 17181 2024-08-31T22:24:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:02.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:02.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:02.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:02.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:02.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:02.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:02.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:02.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:02.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:02.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:02.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:02.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:02.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:02.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=17251], [object_number=118, tag_tid=14, entry_tid=17250], [object_number=117, tag_tid=14, entry_tid=17249], [object_number=116, tag_tid=14, entry_tid=17248]]], state=connected] 2024-08-31T22:24:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=17251], [object_number=118, tag_tid=14, entry_tid=17250], [object_number=117, tag_tid=14, entry_tid=17249], [object_number=116, tag_tid=14, entry_tid=17248]]], state=connected] 2024-08-31T22:24:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=119, tag_tid=14, entry_tid=17251' 2024-08-31T22:24:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=119, tag_tid=14, entry_tid=17251' 2024-08-31T22:24:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=17251' '!=' 'object_number=117, tag_tid=14, entry_tid=17181' 2024-08-31T22:24:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=17251' = 'object_number=117, tag_tid=14, entry_tid=17181' 2024-08-31T22:24:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=119, tag_tid=14, entry_tid=17251' 2024-08-31T22:24:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:02.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:02.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=17251' 2024-08-31T22:24:02.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=17251' 2024-08-31T22:24:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17251 2024-08-31T22:24:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 17251 2024-08-31T22:24:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:03.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:03.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=116, tag_tid=14, entry_tid=17356], [object_number=119, tag_tid=14, entry_tid=17355], [object_number=118, tag_tid=14, entry_tid=17354], [object_number=117, tag_tid=14, entry_tid=17353]]], state=connected] 2024-08-31T22:24:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:03.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=116, tag_tid=14, entry_tid=17356], [object_number=119, tag_tid=14, entry_tid=17355], [object_number=118, tag_tid=14, entry_tid=17354], [object_number=117, tag_tid=14, entry_tid=17353]]], state=connected] 2024-08-31T22:24:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=116, tag_tid=14, entry_tid=17356' 2024-08-31T22:24:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=116, tag_tid=14, entry_tid=17356' 2024-08-31T22:24:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=116, tag_tid=14, entry_tid=17356' '!=' 'object_number=119, tag_tid=14, entry_tid=17251' 2024-08-31T22:24:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=116, tag_tid=14, entry_tid=17356' = 'object_number=119, tag_tid=14, entry_tid=17251' 2024-08-31T22:24:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=116, tag_tid=14, entry_tid=17356' 2024-08-31T22:24:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=116, tag_tid=14, entry_tid=17356' 2024-08-31T22:24:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=116, tag_tid=14, entry_tid=17356' 2024-08-31T22:24:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17356 2024-08-31T22:24:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 17356 2024-08-31T22:24:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=17439], [object_number=118, tag_tid=14, entry_tid=17438], [object_number=117, tag_tid=14, entry_tid=17437], [object_number=116, tag_tid=14, entry_tid=17436]]], state=connected] 2024-08-31T22:24:03.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=17439], [object_number=118, tag_tid=14, entry_tid=17438], [object_number=117, tag_tid=14, entry_tid=17437], [object_number=116, tag_tid=14, entry_tid=17436]]], state=connected] 2024-08-31T22:24:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=119, tag_tid=14, entry_tid=17439' 2024-08-31T22:24:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=119, tag_tid=14, entry_tid=17439' 2024-08-31T22:24:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=17439' '!=' 'object_number=116, tag_tid=14, entry_tid=17356' 2024-08-31T22:24:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=17439' = 'object_number=116, tag_tid=14, entry_tid=17356' 2024-08-31T22:24:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=119, tag_tid=14, entry_tid=17439' 2024-08-31T22:24:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=17439' 2024-08-31T22:24:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=17439' 2024-08-31T22:24:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17439 2024-08-31T22:24:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 17439 2024-08-31T22:24:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:03.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:03.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:04.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=118, tag_tid=14, entry_tid=17518], [object_number=117, tag_tid=14, entry_tid=17517], [object_number=116, tag_tid=14, entry_tid=17516], [object_number=119, tag_tid=14, entry_tid=17515]]], state=connected] 2024-08-31T22:24:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:04.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:04.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:04.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:04.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:04.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=118, tag_tid=14, entry_tid=17518], [object_number=117, tag_tid=14, entry_tid=17517], [object_number=116, tag_tid=14, entry_tid=17516], [object_number=119, tag_tid=14, entry_tid=17515]]], state=connected] 2024-08-31T22:24:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=118, tag_tid=14, entry_tid=17518' 2024-08-31T22:24:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=118, tag_tid=14, entry_tid=17518' 2024-08-31T22:24:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=118, tag_tid=14, entry_tid=17518' '!=' 'object_number=119, tag_tid=14, entry_tid=17439' 2024-08-31T22:24:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=118, tag_tid=14, entry_tid=17518' = 'object_number=119, tag_tid=14, entry_tid=17439' 2024-08-31T22:24:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=118, tag_tid=14, entry_tid=17518' 2024-08-31T22:24:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:04.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=118, tag_tid=14, entry_tid=17518' 2024-08-31T22:24:04.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:04.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=118, tag_tid=14, entry_tid=17518' 2024-08-31T22:24:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17518 2024-08-31T22:24:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 17518 2024-08-31T22:24:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=117, tag_tid=14, entry_tid=17609], [object_number=116, tag_tid=14, entry_tid=17608], [object_number=119, tag_tid=14, entry_tid=17607], [object_number=118, tag_tid=14, entry_tid=17606]]], state=connected] 2024-08-31T22:24:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=117, tag_tid=14, entry_tid=17609], [object_number=116, tag_tid=14, entry_tid=17608], [object_number=119, tag_tid=14, entry_tid=17607], [object_number=118, tag_tid=14, entry_tid=17606]]], state=connected] 2024-08-31T22:24:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=117, tag_tid=14, entry_tid=17609' 2024-08-31T22:24:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=117, tag_tid=14, entry_tid=17609' 2024-08-31T22:24:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=117, tag_tid=14, entry_tid=17609' '!=' 'object_number=118, tag_tid=14, entry_tid=17518' 2024-08-31T22:24:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=117, tag_tid=14, entry_tid=17609' = 'object_number=118, tag_tid=14, entry_tid=17518' 2024-08-31T22:24:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=117, tag_tid=14, entry_tid=17609' 2024-08-31T22:24:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=117, tag_tid=14, entry_tid=17609' 2024-08-31T22:24:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=117, tag_tid=14, entry_tid=17609' 2024-08-31T22:24:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17609 2024-08-31T22:24:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 17609 2024-08-31T22:24:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:05.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:05.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:05.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:05.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:05.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:05.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=118, tag_tid=14, entry_tid=17718], [object_number=117, tag_tid=14, entry_tid=17717], [object_number=116, tag_tid=14, entry_tid=17716], [object_number=119, tag_tid=14, entry_tid=17715]]], state=connected] 2024-08-31T22:24:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=118, tag_tid=14, entry_tid=17718], [object_number=117, tag_tid=14, entry_tid=17717], [object_number=116, tag_tid=14, entry_tid=17716], [object_number=119, tag_tid=14, entry_tid=17715]]], state=connected] 2024-08-31T22:24:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=118, tag_tid=14, entry_tid=17718' 2024-08-31T22:24:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=118, tag_tid=14, entry_tid=17718' 2024-08-31T22:24:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=118, tag_tid=14, entry_tid=17718' '!=' 'object_number=117, tag_tid=14, entry_tid=17609' 2024-08-31T22:24:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=118, tag_tid=14, entry_tid=17718' = 'object_number=117, tag_tid=14, entry_tid=17609' 2024-08-31T22:24:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=118, tag_tid=14, entry_tid=17718' 2024-08-31T22:24:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=118, tag_tid=14, entry_tid=17718' 2024-08-31T22:24:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=118, tag_tid=14, entry_tid=17718' 2024-08-31T22:24:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17718 2024-08-31T22:24:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 17718 2024-08-31T22:24:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:05.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=117, tag_tid=14, entry_tid=17821], [object_number=116, tag_tid=14, entry_tid=17820], [object_number=119, tag_tid=14, entry_tid=17819], [object_number=118, tag_tid=14, entry_tid=17818]]], state=connected] 2024-08-31T22:24:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=117, tag_tid=14, entry_tid=17821], [object_number=116, tag_tid=14, entry_tid=17820], [object_number=119, tag_tid=14, entry_tid=17819], [object_number=118, tag_tid=14, entry_tid=17818]]], state=connected] 2024-08-31T22:24:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=117, tag_tid=14, entry_tid=17821' 2024-08-31T22:24:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=117, tag_tid=14, entry_tid=17821' 2024-08-31T22:24:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=117, tag_tid=14, entry_tid=17821' '!=' 'object_number=118, tag_tid=14, entry_tid=17718' 2024-08-31T22:24:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=117, tag_tid=14, entry_tid=17821' = 'object_number=118, tag_tid=14, entry_tid=17718' 2024-08-31T22:24:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=117, tag_tid=14, entry_tid=17821' 2024-08-31T22:24:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:05.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=117, tag_tid=14, entry_tid=17821' 2024-08-31T22:24:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=117, tag_tid=14, entry_tid=17821' 2024-08-31T22:24:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17821 2024-08-31T22:24:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 17821 2024-08-31T22:24:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:06.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:06.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:06.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:06.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:06.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:06.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:06.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:06.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:06.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:06.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:06.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:06.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:06.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=17927], [object_number=118, tag_tid=14, entry_tid=17926], [object_number=117, tag_tid=14, entry_tid=17925], [object_number=116, tag_tid=14, entry_tid=17924]]], state=connected] 2024-08-31T22:24:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=17927], [object_number=118, tag_tid=14, entry_tid=17926], [object_number=117, tag_tid=14, entry_tid=17925], [object_number=116, tag_tid=14, entry_tid=17924]]], state=connected] 2024-08-31T22:24:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=119, tag_tid=14, entry_tid=17927' 2024-08-31T22:24:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=119, tag_tid=14, entry_tid=17927' 2024-08-31T22:24:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=17927' '!=' 'object_number=117, tag_tid=14, entry_tid=17821' 2024-08-31T22:24:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=17927' = 'object_number=117, tag_tid=14, entry_tid=17821' 2024-08-31T22:24:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=119, tag_tid=14, entry_tid=17927' 2024-08-31T22:24:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=17927' 2024-08-31T22:24:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=17927' 2024-08-31T22:24:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17927 2024-08-31T22:24:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 17927 2024-08-31T22:24:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=117, tag_tid=14, entry_tid=18025], [object_number=116, tag_tid=14, entry_tid=18024], [object_number=119, tag_tid=14, entry_tid=18023], [object_number=118, tag_tid=14, entry_tid=18022]]], state=connected] 2024-08-31T22:24:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=117, tag_tid=14, entry_tid=18025], [object_number=116, tag_tid=14, entry_tid=18024], [object_number=119, tag_tid=14, entry_tid=18023], [object_number=118, tag_tid=14, entry_tid=18022]]], state=connected] 2024-08-31T22:24:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=117, tag_tid=14, entry_tid=18025' 2024-08-31T22:24:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=117, tag_tid=14, entry_tid=18025' 2024-08-31T22:24:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=117, tag_tid=14, entry_tid=18025' '!=' 'object_number=119, tag_tid=14, entry_tid=17927' 2024-08-31T22:24:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=117, tag_tid=14, entry_tid=18025' = 'object_number=119, tag_tid=14, entry_tid=17927' 2024-08-31T22:24:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=117, tag_tid=14, entry_tid=18025' 2024-08-31T22:24:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=117, tag_tid=14, entry_tid=18025' 2024-08-31T22:24:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=117, tag_tid=14, entry_tid=18025' 2024-08-31T22:24:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18025 2024-08-31T22:24:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 18025 2024-08-31T22:24:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:07.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=117, tag_tid=14, entry_tid=18117], [object_number=116, tag_tid=14, entry_tid=18116], [object_number=119, tag_tid=14, entry_tid=18115], [object_number=118, tag_tid=14, entry_tid=18114]]], state=connected] 2024-08-31T22:24:07.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:07.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=117, tag_tid=14, entry_tid=18117], [object_number=116, tag_tid=14, entry_tid=18116], [object_number=119, tag_tid=14, entry_tid=18115], [object_number=118, tag_tid=14, entry_tid=18114]]], state=connected] 2024-08-31T22:24:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=117, tag_tid=14, entry_tid=18117' 2024-08-31T22:24:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=117, tag_tid=14, entry_tid=18117' 2024-08-31T22:24:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=117, tag_tid=14, entry_tid=18117' '!=' 'object_number=117, tag_tid=14, entry_tid=18025' 2024-08-31T22:24:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=117, tag_tid=14, entry_tid=18117' = 'object_number=117, tag_tid=14, entry_tid=18025' 2024-08-31T22:24:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=117, tag_tid=14, entry_tid=18117' 2024-08-31T22:24:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:07.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=117, tag_tid=14, entry_tid=18117' 2024-08-31T22:24:07.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:07.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:07.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:07.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:07.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:07.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=117, tag_tid=14, entry_tid=18117' 2024-08-31T22:24:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18117 2024-08-31T22:24:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 18117 2024-08-31T22:24:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=118, tag_tid=14, entry_tid=18210], [object_number=117, tag_tid=14, entry_tid=18209], [object_number=116, tag_tid=14, entry_tid=18208], [object_number=119, tag_tid=14, entry_tid=18207]]], state=connected] 2024-08-31T22:24:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=118, tag_tid=14, entry_tid=18210], [object_number=117, tag_tid=14, entry_tid=18209], [object_number=116, tag_tid=14, entry_tid=18208], [object_number=119, tag_tid=14, entry_tid=18207]]], state=connected] 2024-08-31T22:24:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=118, tag_tid=14, entry_tid=18210' 2024-08-31T22:24:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=118, tag_tid=14, entry_tid=18210' 2024-08-31T22:24:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=118, tag_tid=14, entry_tid=18210' '!=' 'object_number=117, tag_tid=14, entry_tid=18117' 2024-08-31T22:24:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=118, tag_tid=14, entry_tid=18210' = 'object_number=117, tag_tid=14, entry_tid=18117' 2024-08-31T22:24:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=118, tag_tid=14, entry_tid=18210' 2024-08-31T22:24:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=118, tag_tid=14, entry_tid=18210' 2024-08-31T22:24:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=118, tag_tid=14, entry_tid=18210' 2024-08-31T22:24:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18210 2024-08-31T22:24:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 18210 2024-08-31T22:24:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=116, tag_tid=14, entry_tid=18292], [object_number=119, tag_tid=14, entry_tid=18291], [object_number=118, tag_tid=14, entry_tid=18290], [object_number=117, tag_tid=14, entry_tid=18289]]], state=connected] 2024-08-31T22:24:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:08.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=116, tag_tid=14, entry_tid=18292], [object_number=119, tag_tid=14, entry_tid=18291], [object_number=118, tag_tid=14, entry_tid=18290], [object_number=117, tag_tid=14, entry_tid=18289]]], state=connected] 2024-08-31T22:24:08.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=116, tag_tid=14, entry_tid=18292' 2024-08-31T22:24:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=116, tag_tid=14, entry_tid=18292' 2024-08-31T22:24:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=116, tag_tid=14, entry_tid=18292' '!=' 'object_number=118, tag_tid=14, entry_tid=18210' 2024-08-31T22:24:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=116, tag_tid=14, entry_tid=18292' = 'object_number=118, tag_tid=14, entry_tid=18210' 2024-08-31T22:24:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=116, tag_tid=14, entry_tid=18292' 2024-08-31T22:24:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=116, tag_tid=14, entry_tid=18292' 2024-08-31T22:24:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=116, tag_tid=14, entry_tid=18292' 2024-08-31T22:24:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18292 2024-08-31T22:24:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 18292 2024-08-31T22:24:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=18395], [object_number=118, tag_tid=14, entry_tid=18394], [object_number=117, tag_tid=14, entry_tid=18393], [object_number=116, tag_tid=14, entry_tid=18392]]], state=connected] 2024-08-31T22:24:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=18395], [object_number=118, tag_tid=14, entry_tid=18394], [object_number=117, tag_tid=14, entry_tid=18393], [object_number=116, tag_tid=14, entry_tid=18392]]], state=connected] 2024-08-31T22:24:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=119, tag_tid=14, entry_tid=18395' 2024-08-31T22:24:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=119, tag_tid=14, entry_tid=18395' 2024-08-31T22:24:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=18395' '!=' 'object_number=116, tag_tid=14, entry_tid=18292' 2024-08-31T22:24:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=18395' = 'object_number=116, tag_tid=14, entry_tid=18292' 2024-08-31T22:24:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=119, tag_tid=14, entry_tid=18395' 2024-08-31T22:24:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=18395' 2024-08-31T22:24:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=18395' 2024-08-31T22:24:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18395 2024-08-31T22:24:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 18395 2024-08-31T22:24:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:09.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:09.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=116, tag_tid=14, entry_tid=18488], [object_number=119, tag_tid=14, entry_tid=18487], [object_number=118, tag_tid=14, entry_tid=18486], [object_number=117, tag_tid=14, entry_tid=18485]]], state=connected] 2024-08-31T22:24:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=116, tag_tid=14, entry_tid=18488], [object_number=119, tag_tid=14, entry_tid=18487], [object_number=118, tag_tid=14, entry_tid=18486], [object_number=117, tag_tid=14, entry_tid=18485]]], state=connected] 2024-08-31T22:24:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=116, tag_tid=14, entry_tid=18488' 2024-08-31T22:24:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=116, tag_tid=14, entry_tid=18488' 2024-08-31T22:24:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=116, tag_tid=14, entry_tid=18488' '!=' 'object_number=119, tag_tid=14, entry_tid=18395' 2024-08-31T22:24:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=116, tag_tid=14, entry_tid=18488' = 'object_number=119, tag_tid=14, entry_tid=18395' 2024-08-31T22:24:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=116, tag_tid=14, entry_tid=18488' 2024-08-31T22:24:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=116, tag_tid=14, entry_tid=18488' 2024-08-31T22:24:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=116, tag_tid=14, entry_tid=18488' 2024-08-31T22:24:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18488 2024-08-31T22:24:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 18488 2024-08-31T22:24:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=116, tag_tid=14, entry_tid=18596], [object_number=119, tag_tid=14, entry_tid=18595], [object_number=118, tag_tid=14, entry_tid=18594], [object_number=117, tag_tid=14, entry_tid=18593]]], state=connected] 2024-08-31T22:24:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=116, tag_tid=14, entry_tid=18596], [object_number=119, tag_tid=14, entry_tid=18595], [object_number=118, tag_tid=14, entry_tid=18594], [object_number=117, tag_tid=14, entry_tid=18593]]], state=connected] 2024-08-31T22:24:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=116, tag_tid=14, entry_tid=18596' 2024-08-31T22:24:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=116, tag_tid=14, entry_tid=18596' 2024-08-31T22:24:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=116, tag_tid=14, entry_tid=18596' '!=' 'object_number=116, tag_tid=14, entry_tid=18488' 2024-08-31T22:24:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=116, tag_tid=14, entry_tid=18596' = 'object_number=116, tag_tid=14, entry_tid=18488' 2024-08-31T22:24:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=116, tag_tid=14, entry_tid=18596' 2024-08-31T22:24:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=116, tag_tid=14, entry_tid=18596' 2024-08-31T22:24:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=116, tag_tid=14, entry_tid=18596' 2024-08-31T22:24:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18596 2024-08-31T22:24:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 18596 2024-08-31T22:24:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=117, tag_tid=14, entry_tid=18693], [object_number=116, tag_tid=14, entry_tid=18692], [object_number=119, tag_tid=14, entry_tid=18691], [object_number=118, tag_tid=14, entry_tid=18690]]], state=connected] 2024-08-31T22:24:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=117, tag_tid=14, entry_tid=18693], [object_number=116, tag_tid=14, entry_tid=18692], [object_number=119, tag_tid=14, entry_tid=18691], [object_number=118, tag_tid=14, entry_tid=18690]]], state=connected] 2024-08-31T22:24:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=117, tag_tid=14, entry_tid=18693' 2024-08-31T22:24:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=117, tag_tid=14, entry_tid=18693' 2024-08-31T22:24:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=117, tag_tid=14, entry_tid=18693' '!=' 'object_number=116, tag_tid=14, entry_tid=18596' 2024-08-31T22:24:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=117, tag_tid=14, entry_tid=18693' = 'object_number=116, tag_tid=14, entry_tid=18596' 2024-08-31T22:24:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=117, tag_tid=14, entry_tid=18693' 2024-08-31T22:24:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=117, tag_tid=14, entry_tid=18693' 2024-08-31T22:24:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=117, tag_tid=14, entry_tid=18693' 2024-08-31T22:24:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18693 2024-08-31T22:24:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 18693 2024-08-31T22:24:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:10.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=116, tag_tid=14, entry_tid=18784], [object_number=119, tag_tid=14, entry_tid=18783], [object_number=118, tag_tid=14, entry_tid=18782], [object_number=117, tag_tid=14, entry_tid=18781]]], state=connected] 2024-08-31T22:24:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=116, tag_tid=14, entry_tid=18784], [object_number=119, tag_tid=14, entry_tid=18783], [object_number=118, tag_tid=14, entry_tid=18782], [object_number=117, tag_tid=14, entry_tid=18781]]], state=connected] 2024-08-31T22:24:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=116, tag_tid=14, entry_tid=18784' 2024-08-31T22:24:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=116, tag_tid=14, entry_tid=18784' 2024-08-31T22:24:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=116, tag_tid=14, entry_tid=18784' '!=' 'object_number=117, tag_tid=14, entry_tid=18693' 2024-08-31T22:24:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=116, tag_tid=14, entry_tid=18784' = 'object_number=117, tag_tid=14, entry_tid=18693' 2024-08-31T22:24:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=116, tag_tid=14, entry_tid=18784' 2024-08-31T22:24:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=116, tag_tid=14, entry_tid=18784' 2024-08-31T22:24:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:10.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=116, tag_tid=14, entry_tid=18784' 2024-08-31T22:24:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18784 2024-08-31T22:24:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 18784 2024-08-31T22:24:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:10.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=18879], [object_number=118, tag_tid=14, entry_tid=18878], [object_number=117, tag_tid=14, entry_tid=18877], [object_number=116, tag_tid=14, entry_tid=18876]]], state=connected] 2024-08-31T22:24:11.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:11.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:11.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:11.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:11.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:11.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:11.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:11.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=18879], [object_number=118, tag_tid=14, entry_tid=18878], [object_number=117, tag_tid=14, entry_tid=18877], [object_number=116, tag_tid=14, entry_tid=18876]]], state=connected] 2024-08-31T22:24:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=119, tag_tid=14, entry_tid=18879' 2024-08-31T22:24:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=119, tag_tid=14, entry_tid=18879' 2024-08-31T22:24:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=18879' '!=' 'object_number=116, tag_tid=14, entry_tid=18784' 2024-08-31T22:24:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=18879' = 'object_number=116, tag_tid=14, entry_tid=18784' 2024-08-31T22:24:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=119, tag_tid=14, entry_tid=18879' 2024-08-31T22:24:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=18879' 2024-08-31T22:24:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=18879' 2024-08-31T22:24:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18879 2024-08-31T22:24:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 18879 2024-08-31T22:24:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:11.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:11.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:11.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:11.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=18967], [object_number=118, tag_tid=14, entry_tid=18966], [object_number=117, tag_tid=14, entry_tid=18965], [object_number=116, tag_tid=14, entry_tid=18964]]], state=connected] 2024-08-31T22:24:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=18967], [object_number=118, tag_tid=14, entry_tid=18966], [object_number=117, tag_tid=14, entry_tid=18965], [object_number=116, tag_tid=14, entry_tid=18964]]], state=connected] 2024-08-31T22:24:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=119, tag_tid=14, entry_tid=18967' 2024-08-31T22:24:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=119, tag_tid=14, entry_tid=18967' 2024-08-31T22:24:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=18967' '!=' 'object_number=119, tag_tid=14, entry_tid=18879' 2024-08-31T22:24:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=18967' = 'object_number=119, tag_tid=14, entry_tid=18879' 2024-08-31T22:24:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=119, tag_tid=14, entry_tid=18967' 2024-08-31T22:24:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=18967' 2024-08-31T22:24:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:11.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:11.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=18967' 2024-08-31T22:24:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:11.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18967 2024-08-31T22:24:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 18967 2024-08-31T22:24:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=117, tag_tid=14, entry_tid=19057], [object_number=116, tag_tid=14, entry_tid=19056], [object_number=119, tag_tid=14, entry_tid=19055], [object_number=118, tag_tid=14, entry_tid=19054]]], state=connected] 2024-08-31T22:24:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=117, tag_tid=14, entry_tid=19057], [object_number=116, tag_tid=14, entry_tid=19056], [object_number=119, tag_tid=14, entry_tid=19055], [object_number=118, tag_tid=14, entry_tid=19054]]], state=connected] 2024-08-31T22:24:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=117, tag_tid=14, entry_tid=19057' 2024-08-31T22:24:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=117, tag_tid=14, entry_tid=19057' 2024-08-31T22:24:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=117, tag_tid=14, entry_tid=19057' '!=' 'object_number=119, tag_tid=14, entry_tid=18967' 2024-08-31T22:24:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=117, tag_tid=14, entry_tid=19057' = 'object_number=119, tag_tid=14, entry_tid=18967' 2024-08-31T22:24:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=117, tag_tid=14, entry_tid=19057' 2024-08-31T22:24:12.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:12.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:12.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=117, tag_tid=14, entry_tid=19057' 2024-08-31T22:24:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=117, tag_tid=14, entry_tid=19057' 2024-08-31T22:24:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19057 2024-08-31T22:24:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 19057 2024-08-31T22:24:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:12.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:12.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:12.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:12.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:12.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:12.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:12.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:12.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:12.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:12.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:12.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:12.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:12.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:12.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:12.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:12.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:12.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:12.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:12.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:12.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:12.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:12.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=19143], [object_number=118, tag_tid=14, entry_tid=19142], [object_number=117, tag_tid=14, entry_tid=19141], [object_number=116, tag_tid=14, entry_tid=19140]]], state=connected] 2024-08-31T22:24:12.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=19143], [object_number=118, tag_tid=14, entry_tid=19142], [object_number=117, tag_tid=14, entry_tid=19141], [object_number=116, tag_tid=14, entry_tid=19140]]], state=connected] 2024-08-31T22:24:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=119, tag_tid=14, entry_tid=19143' 2024-08-31T22:24:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=119, tag_tid=14, entry_tid=19143' 2024-08-31T22:24:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=19143' '!=' 'object_number=117, tag_tid=14, entry_tid=19057' 2024-08-31T22:24:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=19143' = 'object_number=117, tag_tid=14, entry_tid=19057' 2024-08-31T22:24:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=119, tag_tid=14, entry_tid=19143' 2024-08-31T22:24:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=19143' 2024-08-31T22:24:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:12.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=19143' 2024-08-31T22:24:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19143 2024-08-31T22:24:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 19143 2024-08-31T22:24:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:12.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:12.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:12.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:12.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:12.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:12.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:12.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:12.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=19239], [object_number=118, tag_tid=14, entry_tid=19238], [object_number=117, tag_tid=14, entry_tid=19237], [object_number=116, tag_tid=14, entry_tid=19236]]], state=connected] 2024-08-31T22:24:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=19239], [object_number=118, tag_tid=14, entry_tid=19238], [object_number=117, tag_tid=14, entry_tid=19237], [object_number=116, tag_tid=14, entry_tid=19236]]], state=connected] 2024-08-31T22:24:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=119, tag_tid=14, entry_tid=19239' 2024-08-31T22:24:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=119, tag_tid=14, entry_tid=19239' 2024-08-31T22:24:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=19239' '!=' 'object_number=119, tag_tid=14, entry_tid=19143' 2024-08-31T22:24:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=19239' = 'object_number=119, tag_tid=14, entry_tid=19143' 2024-08-31T22:24:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=119, tag_tid=14, entry_tid=19239' 2024-08-31T22:24:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:13.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=19239' 2024-08-31T22:24:13.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:13.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:13.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:13.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:13.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:13.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=19239' 2024-08-31T22:24:13.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:13.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:13.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19239 2024-08-31T22:24:13.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 19239 2024-08-31T22:24:13.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:13.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:13.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=116, tag_tid=14, entry_tid=19332], [object_number=119, tag_tid=14, entry_tid=19331], [object_number=118, tag_tid=14, entry_tid=19330], [object_number=117, tag_tid=14, entry_tid=19329]]], state=connected] 2024-08-31T22:24:13.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=116, tag_tid=14, entry_tid=19332], [object_number=119, tag_tid=14, entry_tid=19331], [object_number=118, tag_tid=14, entry_tid=19330], [object_number=117, tag_tid=14, entry_tid=19329]]], state=connected] 2024-08-31T22:24:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=116, tag_tid=14, entry_tid=19332' 2024-08-31T22:24:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=116, tag_tid=14, entry_tid=19332' 2024-08-31T22:24:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=116, tag_tid=14, entry_tid=19332' '!=' 'object_number=119, tag_tid=14, entry_tid=19239' 2024-08-31T22:24:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=116, tag_tid=14, entry_tid=19332' = 'object_number=119, tag_tid=14, entry_tid=19239' 2024-08-31T22:24:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=116, tag_tid=14, entry_tid=19332' 2024-08-31T22:24:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=116, tag_tid=14, entry_tid=19332' 2024-08-31T22:24:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=116, tag_tid=14, entry_tid=19332' 2024-08-31T22:24:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19332 2024-08-31T22:24:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 19332 2024-08-31T22:24:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=118, tag_tid=14, entry_tid=19442], [object_number=117, tag_tid=14, entry_tid=19441], [object_number=116, tag_tid=14, entry_tid=19440], [object_number=119, tag_tid=14, entry_tid=19439]]], state=connected] 2024-08-31T22:24:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=118, tag_tid=14, entry_tid=19442], [object_number=117, tag_tid=14, entry_tid=19441], [object_number=116, tag_tid=14, entry_tid=19440], [object_number=119, tag_tid=14, entry_tid=19439]]], state=connected] 2024-08-31T22:24:14.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=118, tag_tid=14, entry_tid=19442' 2024-08-31T22:24:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=118, tag_tid=14, entry_tid=19442' 2024-08-31T22:24:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=118, tag_tid=14, entry_tid=19442' '!=' 'object_number=116, tag_tid=14, entry_tid=19332' 2024-08-31T22:24:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=118, tag_tid=14, entry_tid=19442' = 'object_number=116, tag_tid=14, entry_tid=19332' 2024-08-31T22:24:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=118, tag_tid=14, entry_tid=19442' 2024-08-31T22:24:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=118, tag_tid=14, entry_tid=19442' 2024-08-31T22:24:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:14.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:14.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:14.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:14.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=118, tag_tid=14, entry_tid=19442' 2024-08-31T22:24:14.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:14.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19442 2024-08-31T22:24:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 19442 2024-08-31T22:24:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:14.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:14.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:14.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:14.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:14.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:14.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:14.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:14.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:14.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:14.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:14.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=116, tag_tid=14, entry_tid=19548], [object_number=119, tag_tid=14, entry_tid=19547], [object_number=118, tag_tid=14, entry_tid=19546], [object_number=117, tag_tid=14, entry_tid=19545]]], state=connected] 2024-08-31T22:24:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=116, tag_tid=14, entry_tid=19548], [object_number=119, tag_tid=14, entry_tid=19547], [object_number=118, tag_tid=14, entry_tid=19546], [object_number=117, tag_tid=14, entry_tid=19545]]], state=connected] 2024-08-31T22:24:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=116, tag_tid=14, entry_tid=19548' 2024-08-31T22:24:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=116, tag_tid=14, entry_tid=19548' 2024-08-31T22:24:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=116, tag_tid=14, entry_tid=19548' '!=' 'object_number=118, tag_tid=14, entry_tid=19442' 2024-08-31T22:24:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=116, tag_tid=14, entry_tid=19548' = 'object_number=118, tag_tid=14, entry_tid=19442' 2024-08-31T22:24:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=116, tag_tid=14, entry_tid=19548' 2024-08-31T22:24:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=116, tag_tid=14, entry_tid=19548' 2024-08-31T22:24:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=116, tag_tid=14, entry_tid=19548' 2024-08-31T22:24:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19548 2024-08-31T22:24:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 19548 2024-08-31T22:24:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=118, tag_tid=14, entry_tid=19650], [object_number=117, tag_tid=14, entry_tid=19649], [object_number=116, tag_tid=14, entry_tid=19648], [object_number=119, tag_tid=14, entry_tid=19647]]], state=connected] 2024-08-31T22:24:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=118, tag_tid=14, entry_tid=19650], [object_number=117, tag_tid=14, entry_tid=19649], [object_number=116, tag_tid=14, entry_tid=19648], [object_number=119, tag_tid=14, entry_tid=19647]]], state=connected] 2024-08-31T22:24:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=118, tag_tid=14, entry_tid=19650' 2024-08-31T22:24:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=118, tag_tid=14, entry_tid=19650' 2024-08-31T22:24:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=118, tag_tid=14, entry_tid=19650' '!=' 'object_number=116, tag_tid=14, entry_tid=19548' 2024-08-31T22:24:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=118, tag_tid=14, entry_tid=19650' = 'object_number=116, tag_tid=14, entry_tid=19548' 2024-08-31T22:24:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=118, tag_tid=14, entry_tid=19650' 2024-08-31T22:24:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=118, tag_tid=14, entry_tid=19650' 2024-08-31T22:24:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:15.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=118, tag_tid=14, entry_tid=19650' 2024-08-31T22:24:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19650 2024-08-31T22:24:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 19650 2024-08-31T22:24:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:15.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=117, tag_tid=14, entry_tid=19757], [object_number=116, tag_tid=14, entry_tid=19756], [object_number=119, tag_tid=14, entry_tid=19755], [object_number=118, tag_tid=14, entry_tid=19754]]], state=connected] 2024-08-31T22:24:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=117, tag_tid=14, entry_tid=19757], [object_number=116, tag_tid=14, entry_tid=19756], [object_number=119, tag_tid=14, entry_tid=19755], [object_number=118, tag_tid=14, entry_tid=19754]]], state=connected] 2024-08-31T22:24:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=117, tag_tid=14, entry_tid=19757' 2024-08-31T22:24:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=117, tag_tid=14, entry_tid=19757' 2024-08-31T22:24:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=117, tag_tid=14, entry_tid=19757' '!=' 'object_number=118, tag_tid=14, entry_tid=19650' 2024-08-31T22:24:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=117, tag_tid=14, entry_tid=19757' = 'object_number=118, tag_tid=14, entry_tid=19650' 2024-08-31T22:24:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=117, tag_tid=14, entry_tid=19757' 2024-08-31T22:24:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=117, tag_tid=14, entry_tid=19757' 2024-08-31T22:24:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=117, tag_tid=14, entry_tid=19757' 2024-08-31T22:24:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19757 2024-08-31T22:24:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 19757 2024-08-31T22:24:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:16.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:16.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:16.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:16.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:16.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:16.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=116, tag_tid=14, entry_tid=19856], [object_number=119, tag_tid=14, entry_tid=19855], [object_number=118, tag_tid=14, entry_tid=19854], [object_number=117, tag_tid=14, entry_tid=19853]]], state=connected] 2024-08-31T22:24:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=116, tag_tid=14, entry_tid=19856], [object_number=119, tag_tid=14, entry_tid=19855], [object_number=118, tag_tid=14, entry_tid=19854], [object_number=117, tag_tid=14, entry_tid=19853]]], state=connected] 2024-08-31T22:24:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=116, tag_tid=14, entry_tid=19856' 2024-08-31T22:24:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=116, tag_tid=14, entry_tid=19856' 2024-08-31T22:24:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=116, tag_tid=14, entry_tid=19856' '!=' 'object_number=117, tag_tid=14, entry_tid=19757' 2024-08-31T22:24:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=116, tag_tid=14, entry_tid=19856' = 'object_number=117, tag_tid=14, entry_tid=19757' 2024-08-31T22:24:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=116, tag_tid=14, entry_tid=19856' 2024-08-31T22:24:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=116, tag_tid=14, entry_tid=19856' 2024-08-31T22:24:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:16.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=116, tag_tid=14, entry_tid=19856' 2024-08-31T22:24:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19856 2024-08-31T22:24:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 19856 2024-08-31T22:24:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:16.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:16.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:16.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:16.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:16.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:16.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:16.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:16.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:16.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:16.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:16.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:16.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:16.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:16.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:16.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:16.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:16.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:16.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:16.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:16.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:16.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=19947], [object_number=118, tag_tid=14, entry_tid=19946], [object_number=117, tag_tid=14, entry_tid=19945], [object_number=116, tag_tid=14, entry_tid=19944]]], state=connected] 2024-08-31T22:24:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=19947], [object_number=118, tag_tid=14, entry_tid=19946], [object_number=117, tag_tid=14, entry_tid=19945], [object_number=116, tag_tid=14, entry_tid=19944]]], state=connected] 2024-08-31T22:24:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=119, tag_tid=14, entry_tid=19947' 2024-08-31T22:24:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=119, tag_tid=14, entry_tid=19947' 2024-08-31T22:24:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=19947' '!=' 'object_number=116, tag_tid=14, entry_tid=19856' 2024-08-31T22:24:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=19947' = 'object_number=116, tag_tid=14, entry_tid=19856' 2024-08-31T22:24:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=119, tag_tid=14, entry_tid=19947' 2024-08-31T22:24:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=19947' 2024-08-31T22:24:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=19947' 2024-08-31T22:24:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=19947 2024-08-31T22:24:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 19947 2024-08-31T22:24:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:16.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:17.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=118, tag_tid=14, entry_tid=20034], [object_number=117, tag_tid=14, entry_tid=20033], [object_number=116, tag_tid=14, entry_tid=20032], [object_number=119, tag_tid=14, entry_tid=20031]]], state=connected] 2024-08-31T22:24:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:17.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:17.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:17.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:17.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:17.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=118, tag_tid=14, entry_tid=20034], [object_number=117, tag_tid=14, entry_tid=20033], [object_number=116, tag_tid=14, entry_tid=20032], [object_number=119, tag_tid=14, entry_tid=20031]]], state=connected] 2024-08-31T22:24:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=118, tag_tid=14, entry_tid=20034' 2024-08-31T22:24:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=118, tag_tid=14, entry_tid=20034' 2024-08-31T22:24:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=118, tag_tid=14, entry_tid=20034' '!=' 'object_number=119, tag_tid=14, entry_tid=19947' 2024-08-31T22:24:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=118, tag_tid=14, entry_tid=20034' = 'object_number=119, tag_tid=14, entry_tid=19947' 2024-08-31T22:24:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=118, tag_tid=14, entry_tid=20034' 2024-08-31T22:24:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=118, tag_tid=14, entry_tid=20034' 2024-08-31T22:24:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=118, tag_tid=14, entry_tid=20034' 2024-08-31T22:24:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20034 2024-08-31T22:24:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 20034 2024-08-31T22:24:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:17.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:17.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:17.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:17.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=117, tag_tid=14, entry_tid=20121], [object_number=116, tag_tid=14, entry_tid=20120], [object_number=119, tag_tid=14, entry_tid=20119], [object_number=118, tag_tid=14, entry_tid=20118]]], state=connected] 2024-08-31T22:24:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=117, tag_tid=14, entry_tid=20121], [object_number=116, tag_tid=14, entry_tid=20120], [object_number=119, tag_tid=14, entry_tid=20119], [object_number=118, tag_tid=14, entry_tid=20118]]], state=connected] 2024-08-31T22:24:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=117, tag_tid=14, entry_tid=20121' 2024-08-31T22:24:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=117, tag_tid=14, entry_tid=20121' 2024-08-31T22:24:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=117, tag_tid=14, entry_tid=20121' '!=' 'object_number=118, tag_tid=14, entry_tid=20034' 2024-08-31T22:24:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=117, tag_tid=14, entry_tid=20121' = 'object_number=118, tag_tid=14, entry_tid=20034' 2024-08-31T22:24:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=117, tag_tid=14, entry_tid=20121' 2024-08-31T22:24:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=117, tag_tid=14, entry_tid=20121' 2024-08-31T22:24:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=117, tag_tid=14, entry_tid=20121' 2024-08-31T22:24:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20121 2024-08-31T22:24:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 20121 2024-08-31T22:24:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:18.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=20219], [object_number=118, tag_tid=14, entry_tid=20218], [object_number=117, tag_tid=14, entry_tid=20217], [object_number=116, tag_tid=14, entry_tid=20216]]], state=connected] 2024-08-31T22:24:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=20219], [object_number=118, tag_tid=14, entry_tid=20218], [object_number=117, tag_tid=14, entry_tid=20217], [object_number=116, tag_tid=14, entry_tid=20216]]], state=connected] 2024-08-31T22:24:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=119, tag_tid=14, entry_tid=20219' 2024-08-31T22:24:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=119, tag_tid=14, entry_tid=20219' 2024-08-31T22:24:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=20219' '!=' 'object_number=117, tag_tid=14, entry_tid=20121' 2024-08-31T22:24:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=20219' = 'object_number=117, tag_tid=14, entry_tid=20121' 2024-08-31T22:24:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=119, tag_tid=14, entry_tid=20219' 2024-08-31T22:24:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=20219' 2024-08-31T22:24:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=20219' 2024-08-31T22:24:18.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:18.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20219 2024-08-31T22:24:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 20219 2024-08-31T22:24:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:18.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:18.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=118, tag_tid=14, entry_tid=20342], [object_number=117, tag_tid=14, entry_tid=20341], [object_number=116, tag_tid=14, entry_tid=20340], [object_number=119, tag_tid=14, entry_tid=20339]]], state=connected] 2024-08-31T22:24:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=118, tag_tid=14, entry_tid=20342], [object_number=117, tag_tid=14, entry_tid=20341], [object_number=116, tag_tid=14, entry_tid=20340], [object_number=119, tag_tid=14, entry_tid=20339]]], state=connected] 2024-08-31T22:24:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=118, tag_tid=14, entry_tid=20342' 2024-08-31T22:24:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=118, tag_tid=14, entry_tid=20342' 2024-08-31T22:24:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=118, tag_tid=14, entry_tid=20342' '!=' 'object_number=119, tag_tid=14, entry_tid=20219' 2024-08-31T22:24:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=118, tag_tid=14, entry_tid=20342' = 'object_number=119, tag_tid=14, entry_tid=20219' 2024-08-31T22:24:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=118, tag_tid=14, entry_tid=20342' 2024-08-31T22:24:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=118, tag_tid=14, entry_tid=20342' 2024-08-31T22:24:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=118, tag_tid=14, entry_tid=20342' 2024-08-31T22:24:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20342 2024-08-31T22:24:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 20342 2024-08-31T22:24:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:19.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:19.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:19.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:19.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:19.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=116, tag_tid=14, entry_tid=20428], [object_number=119, tag_tid=14, entry_tid=20427], [object_number=118, tag_tid=14, entry_tid=20426], [object_number=117, tag_tid=14, entry_tid=20425]]], state=connected] 2024-08-31T22:24:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:19.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:19.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:19.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:19.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:19.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=116, tag_tid=14, entry_tid=20428], [object_number=119, tag_tid=14, entry_tid=20427], [object_number=118, tag_tid=14, entry_tid=20426], [object_number=117, tag_tid=14, entry_tid=20425]]], state=connected] 2024-08-31T22:24:19.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=116, tag_tid=14, entry_tid=20428' 2024-08-31T22:24:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=116, tag_tid=14, entry_tid=20428' 2024-08-31T22:24:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=116, tag_tid=14, entry_tid=20428' '!=' 'object_number=118, tag_tid=14, entry_tid=20342' 2024-08-31T22:24:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=116, tag_tid=14, entry_tid=20428' = 'object_number=118, tag_tid=14, entry_tid=20342' 2024-08-31T22:24:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=116, tag_tid=14, entry_tid=20428' 2024-08-31T22:24:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:19.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:19.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:19.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=116, tag_tid=14, entry_tid=20428' 2024-08-31T22:24:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:19.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:19.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=116, tag_tid=14, entry_tid=20428' 2024-08-31T22:24:19.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:19.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20428 2024-08-31T22:24:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 20428 2024-08-31T22:24:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:19.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:19.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:19.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=20535], [object_number=118, tag_tid=14, entry_tid=20534], [object_number=117, tag_tid=14, entry_tid=20533], [object_number=116, tag_tid=14, entry_tid=20532]]], state=connected] 2024-08-31T22:24:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 29 2024-08-31T22:24:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=119, tag_tid=14, entry_tid=20535], [object_number=118, tag_tid=14, entry_tid=20534], [object_number=117, tag_tid=14, entry_tid=20533], [object_number=116, tag_tid=14, entry_tid=20532]]], state=connected] 2024-08-31T22:24:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=119, tag_tid=14, entry_tid=20535' 2024-08-31T22:24:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=119, tag_tid=14, entry_tid=20535' 2024-08-31T22:24:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=20535' '!=' 'object_number=116, tag_tid=14, entry_tid=20428' 2024-08-31T22:24:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=119, tag_tid=14, entry_tid=20535' = 'object_number=116, tag_tid=14, entry_tid=20428' 2024-08-31T22:24:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=119, tag_tid=14, entry_tid=20535' 2024-08-31T22:24:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=20535' 2024-08-31T22:24:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=119, tag_tid=14, entry_tid=20535' 2024-08-31T22:24:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20535 2024-08-31T22:24:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 20535 2024-08-31T22:24:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=121, tag_tid=14, entry_tid=20621], [object_number=120, tag_tid=14, entry_tid=20620], [object_number=123, tag_tid=14, entry_tid=20619], [object_number=122, tag_tid=14, entry_tid=20618]]], state=connected] 2024-08-31T22:24:20.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=121, tag_tid=14, entry_tid=20621], [object_number=120, tag_tid=14, entry_tid=20620], [object_number=123, tag_tid=14, entry_tid=20619], [object_number=122, tag_tid=14, entry_tid=20618]]], state=connected] 2024-08-31T22:24:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=121, tag_tid=14, entry_tid=20621' 2024-08-31T22:24:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=121, tag_tid=14, entry_tid=20621' 2024-08-31T22:24:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=121, tag_tid=14, entry_tid=20621' '!=' 'object_number=119, tag_tid=14, entry_tid=20535' 2024-08-31T22:24:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=121, tag_tid=14, entry_tid=20621' = 'object_number=119, tag_tid=14, entry_tid=20535' 2024-08-31T22:24:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=121, tag_tid=14, entry_tid=20621' 2024-08-31T22:24:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=14, entry_tid=20621' 2024-08-31T22:24:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:20.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:20.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:20.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=14, entry_tid=20621' 2024-08-31T22:24:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20621 2024-08-31T22:24:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 20621 2024-08-31T22:24:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:20.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:20.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:20.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:20.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=122, tag_tid=14, entry_tid=20718], [object_number=121, tag_tid=14, entry_tid=20717], [object_number=120, tag_tid=14, entry_tid=20716], [object_number=123, tag_tid=14, entry_tid=20715]]], state=connected] 2024-08-31T22:24:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:20.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:20.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:20.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:20.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:20.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:20.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:20.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:20.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:20.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:20.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:20.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:20.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=122, tag_tid=14, entry_tid=20718], [object_number=121, tag_tid=14, entry_tid=20717], [object_number=120, tag_tid=14, entry_tid=20716], [object_number=123, tag_tid=14, entry_tid=20715]]], state=connected] 2024-08-31T22:24:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=122, tag_tid=14, entry_tid=20718' 2024-08-31T22:24:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=122, tag_tid=14, entry_tid=20718' 2024-08-31T22:24:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=122, tag_tid=14, entry_tid=20718' '!=' 'object_number=121, tag_tid=14, entry_tid=20621' 2024-08-31T22:24:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=122, tag_tid=14, entry_tid=20718' = 'object_number=121, tag_tid=14, entry_tid=20621' 2024-08-31T22:24:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=122, tag_tid=14, entry_tid=20718' 2024-08-31T22:24:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=122, tag_tid=14, entry_tid=20718' 2024-08-31T22:24:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=122, tag_tid=14, entry_tid=20718' 2024-08-31T22:24:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20718 2024-08-31T22:24:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 20718 2024-08-31T22:24:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:21.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:21.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:21.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:21.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:21.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=123, tag_tid=14, entry_tid=20815], [object_number=122, tag_tid=14, entry_tid=20814], [object_number=121, tag_tid=14, entry_tid=20813], [object_number=120, tag_tid=14, entry_tid=20812]]], state=connected] 2024-08-31T22:24:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=123, tag_tid=14, entry_tid=20815], [object_number=122, tag_tid=14, entry_tid=20814], [object_number=121, tag_tid=14, entry_tid=20813], [object_number=120, tag_tid=14, entry_tid=20812]]], state=connected] 2024-08-31T22:24:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=123, tag_tid=14, entry_tid=20815' 2024-08-31T22:24:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=123, tag_tid=14, entry_tid=20815' 2024-08-31T22:24:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=123, tag_tid=14, entry_tid=20815' '!=' 'object_number=122, tag_tid=14, entry_tid=20718' 2024-08-31T22:24:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=123, tag_tid=14, entry_tid=20815' = 'object_number=122, tag_tid=14, entry_tid=20718' 2024-08-31T22:24:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=123, tag_tid=14, entry_tid=20815' 2024-08-31T22:24:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=123, tag_tid=14, entry_tid=20815' 2024-08-31T22:24:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=123, tag_tid=14, entry_tid=20815' 2024-08-31T22:24:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20815 2024-08-31T22:24:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 20815 2024-08-31T22:24:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=123, tag_tid=14, entry_tid=20907], [object_number=122, tag_tid=14, entry_tid=20906], [object_number=121, tag_tid=14, entry_tid=20905], [object_number=120, tag_tid=14, entry_tid=20904]]], state=connected] 2024-08-31T22:24:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=123, tag_tid=14, entry_tid=20907], [object_number=122, tag_tid=14, entry_tid=20906], [object_number=121, tag_tid=14, entry_tid=20905], [object_number=120, tag_tid=14, entry_tid=20904]]], state=connected] 2024-08-31T22:24:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=123, tag_tid=14, entry_tid=20907' 2024-08-31T22:24:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=123, tag_tid=14, entry_tid=20907' 2024-08-31T22:24:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=123, tag_tid=14, entry_tid=20907' '!=' 'object_number=123, tag_tid=14, entry_tid=20815' 2024-08-31T22:24:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=123, tag_tid=14, entry_tid=20907' = 'object_number=123, tag_tid=14, entry_tid=20815' 2024-08-31T22:24:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=123, tag_tid=14, entry_tid=20907' 2024-08-31T22:24:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=123, tag_tid=14, entry_tid=20907' 2024-08-31T22:24:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=123, tag_tid=14, entry_tid=20907' 2024-08-31T22:24:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20907 2024-08-31T22:24:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 20907 2024-08-31T22:24:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=121, tag_tid=14, entry_tid=20997], [object_number=120, tag_tid=14, entry_tid=20996], [object_number=123, tag_tid=14, entry_tid=20995], [object_number=122, tag_tid=14, entry_tid=20994]]], state=connected] 2024-08-31T22:24:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=121, tag_tid=14, entry_tid=20997], [object_number=120, tag_tid=14, entry_tid=20996], [object_number=123, tag_tid=14, entry_tid=20995], [object_number=122, tag_tid=14, entry_tid=20994]]], state=connected] 2024-08-31T22:24:22.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=121, tag_tid=14, entry_tid=20997' 2024-08-31T22:24:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=121, tag_tid=14, entry_tid=20997' 2024-08-31T22:24:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=121, tag_tid=14, entry_tid=20997' '!=' 'object_number=123, tag_tid=14, entry_tid=20907' 2024-08-31T22:24:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=121, tag_tid=14, entry_tid=20997' = 'object_number=123, tag_tid=14, entry_tid=20907' 2024-08-31T22:24:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=121, tag_tid=14, entry_tid=20997' 2024-08-31T22:24:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:22.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:22.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=14, entry_tid=20997' 2024-08-31T22:24:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=14, entry_tid=20997' 2024-08-31T22:24:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=20997 2024-08-31T22:24:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 20997 2024-08-31T22:24:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:22.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:22.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:22.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:22.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:22.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:22.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:22.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:22.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=123, tag_tid=14, entry_tid=21083], [object_number=122, tag_tid=14, entry_tid=21082], [object_number=121, tag_tid=14, entry_tid=21081], [object_number=120, tag_tid=14, entry_tid=21080]]], state=connected] 2024-08-31T22:24:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=123, tag_tid=14, entry_tid=21083], [object_number=122, tag_tid=14, entry_tid=21082], [object_number=121, tag_tid=14, entry_tid=21081], [object_number=120, tag_tid=14, entry_tid=21080]]], state=connected] 2024-08-31T22:24:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=123, tag_tid=14, entry_tid=21083' 2024-08-31T22:24:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=123, tag_tid=14, entry_tid=21083' 2024-08-31T22:24:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=123, tag_tid=14, entry_tid=21083' '!=' 'object_number=121, tag_tid=14, entry_tid=20997' 2024-08-31T22:24:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=123, tag_tid=14, entry_tid=21083' = 'object_number=121, tag_tid=14, entry_tid=20997' 2024-08-31T22:24:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=123, tag_tid=14, entry_tid=21083' 2024-08-31T22:24:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=123, tag_tid=14, entry_tid=21083' 2024-08-31T22:24:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=123, tag_tid=14, entry_tid=21083' 2024-08-31T22:24:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21083 2024-08-31T22:24:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 21083 2024-08-31T22:24:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=122, tag_tid=14, entry_tid=21174], [object_number=121, tag_tid=14, entry_tid=21173], [object_number=120, tag_tid=14, entry_tid=21172], [object_number=123, tag_tid=14, entry_tid=21171]]], state=connected] 2024-08-31T22:24:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=122, tag_tid=14, entry_tid=21174], [object_number=121, tag_tid=14, entry_tid=21173], [object_number=120, tag_tid=14, entry_tid=21172], [object_number=123, tag_tid=14, entry_tid=21171]]], state=connected] 2024-08-31T22:24:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=122, tag_tid=14, entry_tid=21174' 2024-08-31T22:24:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=122, tag_tid=14, entry_tid=21174' 2024-08-31T22:24:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=122, tag_tid=14, entry_tid=21174' '!=' 'object_number=123, tag_tid=14, entry_tid=21083' 2024-08-31T22:24:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=122, tag_tid=14, entry_tid=21174' = 'object_number=123, tag_tid=14, entry_tid=21083' 2024-08-31T22:24:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=122, tag_tid=14, entry_tid=21174' 2024-08-31T22:24:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=122, tag_tid=14, entry_tid=21174' 2024-08-31T22:24:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=122, tag_tid=14, entry_tid=21174' 2024-08-31T22:24:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21174 2024-08-31T22:24:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 21174 2024-08-31T22:24:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:23.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:23.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=120, tag_tid=14, entry_tid=21256], [object_number=123, tag_tid=14, entry_tid=21255], [object_number=122, tag_tid=14, entry_tid=21254], [object_number=121, tag_tid=14, entry_tid=21253]]], state=connected] 2024-08-31T22:24:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=120, tag_tid=14, entry_tid=21256], [object_number=123, tag_tid=14, entry_tid=21255], [object_number=122, tag_tid=14, entry_tid=21254], [object_number=121, tag_tid=14, entry_tid=21253]]], state=connected] 2024-08-31T22:24:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=120, tag_tid=14, entry_tid=21256' 2024-08-31T22:24:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=120, tag_tid=14, entry_tid=21256' 2024-08-31T22:24:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=120, tag_tid=14, entry_tid=21256' '!=' 'object_number=122, tag_tid=14, entry_tid=21174' 2024-08-31T22:24:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=120, tag_tid=14, entry_tid=21256' = 'object_number=122, tag_tid=14, entry_tid=21174' 2024-08-31T22:24:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=120, tag_tid=14, entry_tid=21256' 2024-08-31T22:24:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=120, tag_tid=14, entry_tid=21256' 2024-08-31T22:24:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=120, tag_tid=14, entry_tid=21256' 2024-08-31T22:24:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21256 2024-08-31T22:24:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 21256 2024-08-31T22:24:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:24.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:24.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=120, tag_tid=14, entry_tid=21336], [object_number=123, tag_tid=14, entry_tid=21335], [object_number=122, tag_tid=14, entry_tid=21334], [object_number=121, tag_tid=14, entry_tid=21333]]], state=connected] 2024-08-31T22:24:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=120, tag_tid=14, entry_tid=21336], [object_number=123, tag_tid=14, entry_tid=21335], [object_number=122, tag_tid=14, entry_tid=21334], [object_number=121, tag_tid=14, entry_tid=21333]]], state=connected] 2024-08-31T22:24:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=120, tag_tid=14, entry_tid=21336' 2024-08-31T22:24:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=120, tag_tid=14, entry_tid=21336' 2024-08-31T22:24:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=120, tag_tid=14, entry_tid=21336' '!=' 'object_number=120, tag_tid=14, entry_tid=21256' 2024-08-31T22:24:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=120, tag_tid=14, entry_tid=21336' = 'object_number=120, tag_tid=14, entry_tid=21256' 2024-08-31T22:24:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=120, tag_tid=14, entry_tid=21336' 2024-08-31T22:24:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=120, tag_tid=14, entry_tid=21336' 2024-08-31T22:24:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=120, tag_tid=14, entry_tid=21336' 2024-08-31T22:24:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21336 2024-08-31T22:24:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 21336 2024-08-31T22:24:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:24.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:24.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:24.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:24.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:24.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:24.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:24.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:24.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:24.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=123, tag_tid=14, entry_tid=21423], [object_number=122, tag_tid=14, entry_tid=21422], [object_number=121, tag_tid=14, entry_tid=21421], [object_number=120, tag_tid=14, entry_tid=21420]]], state=connected] 2024-08-31T22:24:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=123, tag_tid=14, entry_tid=21423], [object_number=122, tag_tid=14, entry_tid=21422], [object_number=121, tag_tid=14, entry_tid=21421], [object_number=120, tag_tid=14, entry_tid=21420]]], state=connected] 2024-08-31T22:24:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=123, tag_tid=14, entry_tid=21423' 2024-08-31T22:24:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=123, tag_tid=14, entry_tid=21423' 2024-08-31T22:24:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=123, tag_tid=14, entry_tid=21423' '!=' 'object_number=120, tag_tid=14, entry_tid=21336' 2024-08-31T22:24:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=123, tag_tid=14, entry_tid=21423' = 'object_number=120, tag_tid=14, entry_tid=21336' 2024-08-31T22:24:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=123, tag_tid=14, entry_tid=21423' 2024-08-31T22:24:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=123, tag_tid=14, entry_tid=21423' 2024-08-31T22:24:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:24.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=123, tag_tid=14, entry_tid=21423' 2024-08-31T22:24:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21423 2024-08-31T22:24:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 21423 2024-08-31T22:24:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=122, tag_tid=14, entry_tid=21510], [object_number=121, tag_tid=14, entry_tid=21509], [object_number=120, tag_tid=14, entry_tid=21508], [object_number=123, tag_tid=14, entry_tid=21507]]], state=connected] 2024-08-31T22:24:25.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=122, tag_tid=14, entry_tid=21510], [object_number=121, tag_tid=14, entry_tid=21509], [object_number=120, tag_tid=14, entry_tid=21508], [object_number=123, tag_tid=14, entry_tid=21507]]], state=connected] 2024-08-31T22:24:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=122, tag_tid=14, entry_tid=21510' 2024-08-31T22:24:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=122, tag_tid=14, entry_tid=21510' 2024-08-31T22:24:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=122, tag_tid=14, entry_tid=21510' '!=' 'object_number=123, tag_tid=14, entry_tid=21423' 2024-08-31T22:24:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=122, tag_tid=14, entry_tid=21510' = 'object_number=123, tag_tid=14, entry_tid=21423' 2024-08-31T22:24:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=122, tag_tid=14, entry_tid=21510' 2024-08-31T22:24:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=122, tag_tid=14, entry_tid=21510' 2024-08-31T22:24:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=122, tag_tid=14, entry_tid=21510' 2024-08-31T22:24:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21510 2024-08-31T22:24:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 21510 2024-08-31T22:24:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:25.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:25.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:25.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=122, tag_tid=14, entry_tid=21598], [object_number=121, tag_tid=14, entry_tid=21597], [object_number=120, tag_tid=14, entry_tid=21596], [object_number=123, tag_tid=14, entry_tid=21595]]], state=connected] 2024-08-31T22:24:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:25.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:25.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:25.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:25.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:25.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:25.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=122, tag_tid=14, entry_tid=21598], [object_number=121, tag_tid=14, entry_tid=21597], [object_number=120, tag_tid=14, entry_tid=21596], [object_number=123, tag_tid=14, entry_tid=21595]]], state=connected] 2024-08-31T22:24:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=122, tag_tid=14, entry_tid=21598' 2024-08-31T22:24:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=122, tag_tid=14, entry_tid=21598' 2024-08-31T22:24:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=122, tag_tid=14, entry_tid=21598' '!=' 'object_number=122, tag_tid=14, entry_tid=21510' 2024-08-31T22:24:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=122, tag_tid=14, entry_tid=21598' = 'object_number=122, tag_tid=14, entry_tid=21510' 2024-08-31T22:24:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=122, tag_tid=14, entry_tid=21598' 2024-08-31T22:24:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=122, tag_tid=14, entry_tid=21598' 2024-08-31T22:24:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=122, tag_tid=14, entry_tid=21598' 2024-08-31T22:24:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21598 2024-08-31T22:24:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 21598 2024-08-31T22:24:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:26.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:26.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=123, tag_tid=14, entry_tid=21683], [object_number=122, tag_tid=14, entry_tid=21682], [object_number=121, tag_tid=14, entry_tid=21681], [object_number=120, tag_tid=14, entry_tid=21680]]], state=connected] 2024-08-31T22:24:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=123, tag_tid=14, entry_tid=21683], [object_number=122, tag_tid=14, entry_tid=21682], [object_number=121, tag_tid=14, entry_tid=21681], [object_number=120, tag_tid=14, entry_tid=21680]]], state=connected] 2024-08-31T22:24:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=123, tag_tid=14, entry_tid=21683' 2024-08-31T22:24:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=123, tag_tid=14, entry_tid=21683' 2024-08-31T22:24:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=123, tag_tid=14, entry_tid=21683' '!=' 'object_number=122, tag_tid=14, entry_tid=21598' 2024-08-31T22:24:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=123, tag_tid=14, entry_tid=21683' = 'object_number=122, tag_tid=14, entry_tid=21598' 2024-08-31T22:24:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=123, tag_tid=14, entry_tid=21683' 2024-08-31T22:24:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=123, tag_tid=14, entry_tid=21683' 2024-08-31T22:24:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=123, tag_tid=14, entry_tid=21683' 2024-08-31T22:24:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21683 2024-08-31T22:24:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 21683 2024-08-31T22:24:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=122, tag_tid=14, entry_tid=21766], [object_number=121, tag_tid=14, entry_tid=21765], [object_number=120, tag_tid=14, entry_tid=21764], [object_number=123, tag_tid=14, entry_tid=21763]]], state=connected] 2024-08-31T22:24:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:26.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:26.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:26.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=122, tag_tid=14, entry_tid=21766], [object_number=121, tag_tid=14, entry_tid=21765], [object_number=120, tag_tid=14, entry_tid=21764], [object_number=123, tag_tid=14, entry_tid=21763]]], state=connected] 2024-08-31T22:24:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=122, tag_tid=14, entry_tid=21766' 2024-08-31T22:24:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=122, tag_tid=14, entry_tid=21766' 2024-08-31T22:24:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=122, tag_tid=14, entry_tid=21766' '!=' 'object_number=123, tag_tid=14, entry_tid=21683' 2024-08-31T22:24:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=122, tag_tid=14, entry_tid=21766' = 'object_number=123, tag_tid=14, entry_tid=21683' 2024-08-31T22:24:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=122, tag_tid=14, entry_tid=21766' 2024-08-31T22:24:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=122, tag_tid=14, entry_tid=21766' 2024-08-31T22:24:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=122, tag_tid=14, entry_tid=21766' 2024-08-31T22:24:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21766 2024-08-31T22:24:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 21766 2024-08-31T22:24:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=121, tag_tid=14, entry_tid=21853], [object_number=120, tag_tid=14, entry_tid=21852], [object_number=123, tag_tid=14, entry_tid=21851], [object_number=122, tag_tid=14, entry_tid=21850]]], state=connected] 2024-08-31T22:24:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=121, tag_tid=14, entry_tid=21853], [object_number=120, tag_tid=14, entry_tid=21852], [object_number=123, tag_tid=14, entry_tid=21851], [object_number=122, tag_tid=14, entry_tid=21850]]], state=connected] 2024-08-31T22:24:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=121, tag_tid=14, entry_tid=21853' 2024-08-31T22:24:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=121, tag_tid=14, entry_tid=21853' 2024-08-31T22:24:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=121, tag_tid=14, entry_tid=21853' '!=' 'object_number=122, tag_tid=14, entry_tid=21766' 2024-08-31T22:24:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=121, tag_tid=14, entry_tid=21853' = 'object_number=122, tag_tid=14, entry_tid=21766' 2024-08-31T22:24:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=121, tag_tid=14, entry_tid=21853' 2024-08-31T22:24:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=14, entry_tid=21853' 2024-08-31T22:24:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=14, entry_tid=21853' 2024-08-31T22:24:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21853 2024-08-31T22:24:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 21853 2024-08-31T22:24:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=121, tag_tid=14, entry_tid=21945], [object_number=120, tag_tid=14, entry_tid=21944], [object_number=123, tag_tid=14, entry_tid=21943], [object_number=122, tag_tid=14, entry_tid=21942]]], state=connected] 2024-08-31T22:24:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:27.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:27.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=121, tag_tid=14, entry_tid=21945], [object_number=120, tag_tid=14, entry_tid=21944], [object_number=123, tag_tid=14, entry_tid=21943], [object_number=122, tag_tid=14, entry_tid=21942]]], state=connected] 2024-08-31T22:24:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=121, tag_tid=14, entry_tid=21945' 2024-08-31T22:24:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=121, tag_tid=14, entry_tid=21945' 2024-08-31T22:24:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=121, tag_tid=14, entry_tid=21945' '!=' 'object_number=121, tag_tid=14, entry_tid=21853' 2024-08-31T22:24:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=121, tag_tid=14, entry_tid=21945' = 'object_number=121, tag_tid=14, entry_tid=21853' 2024-08-31T22:24:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=121, tag_tid=14, entry_tid=21945' 2024-08-31T22:24:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=14, entry_tid=21945' 2024-08-31T22:24:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=14, entry_tid=21945' 2024-08-31T22:24:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=21945 2024-08-31T22:24:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 21945 2024-08-31T22:24:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:28.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=123, tag_tid=14, entry_tid=22031], [object_number=122, tag_tid=14, entry_tid=22030], [object_number=121, tag_tid=14, entry_tid=22029], [object_number=120, tag_tid=14, entry_tid=22028]]], state=connected] 2024-08-31T22:24:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:28.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:28.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:28.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:28.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:28.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=123, tag_tid=14, entry_tid=22031], [object_number=122, tag_tid=14, entry_tid=22030], [object_number=121, tag_tid=14, entry_tid=22029], [object_number=120, tag_tid=14, entry_tid=22028]]], state=connected] 2024-08-31T22:24:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=123, tag_tid=14, entry_tid=22031' 2024-08-31T22:24:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=123, tag_tid=14, entry_tid=22031' 2024-08-31T22:24:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=123, tag_tid=14, entry_tid=22031' '!=' 'object_number=121, tag_tid=14, entry_tid=21945' 2024-08-31T22:24:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=123, tag_tid=14, entry_tid=22031' = 'object_number=121, tag_tid=14, entry_tid=21945' 2024-08-31T22:24:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=123, tag_tid=14, entry_tid=22031' 2024-08-31T22:24:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=123, tag_tid=14, entry_tid=22031' 2024-08-31T22:24:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=123, tag_tid=14, entry_tid=22031' 2024-08-31T22:24:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=22031 2024-08-31T22:24:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 22031 2024-08-31T22:24:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:28.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:28.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=123, tag_tid=14, entry_tid=22119], [object_number=122, tag_tid=14, entry_tid=22118], [object_number=121, tag_tid=14, entry_tid=22117], [object_number=120, tag_tid=14, entry_tid=22116]]], state=connected] 2024-08-31T22:24:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=123, tag_tid=14, entry_tid=22119], [object_number=122, tag_tid=14, entry_tid=22118], [object_number=121, tag_tid=14, entry_tid=22117], [object_number=120, tag_tid=14, entry_tid=22116]]], state=connected] 2024-08-31T22:24:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=123, tag_tid=14, entry_tid=22119' 2024-08-31T22:24:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=123, tag_tid=14, entry_tid=22119' 2024-08-31T22:24:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=123, tag_tid=14, entry_tid=22119' '!=' 'object_number=123, tag_tid=14, entry_tid=22031' 2024-08-31T22:24:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=123, tag_tid=14, entry_tid=22119' = 'object_number=123, tag_tid=14, entry_tid=22031' 2024-08-31T22:24:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=123, tag_tid=14, entry_tid=22119' 2024-08-31T22:24:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=123, tag_tid=14, entry_tid=22119' 2024-08-31T22:24:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=123, tag_tid=14, entry_tid=22119' 2024-08-31T22:24:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=22119 2024-08-31T22:24:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 22119 2024-08-31T22:24:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:29.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:29.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:29.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=120, tag_tid=14, entry_tid=22204], [object_number=123, tag_tid=14, entry_tid=22203], [object_number=122, tag_tid=14, entry_tid=22202], [object_number=121, tag_tid=14, entry_tid=22201]]], state=connected] 2024-08-31T22:24:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=120, tag_tid=14, entry_tid=22204], [object_number=123, tag_tid=14, entry_tid=22203], [object_number=122, tag_tid=14, entry_tid=22202], [object_number=121, tag_tid=14, entry_tid=22201]]], state=connected] 2024-08-31T22:24:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=120, tag_tid=14, entry_tid=22204' 2024-08-31T22:24:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=120, tag_tid=14, entry_tid=22204' 2024-08-31T22:24:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=120, tag_tid=14, entry_tid=22204' '!=' 'object_number=123, tag_tid=14, entry_tid=22119' 2024-08-31T22:24:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=120, tag_tid=14, entry_tid=22204' = 'object_number=123, tag_tid=14, entry_tid=22119' 2024-08-31T22:24:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=120, tag_tid=14, entry_tid=22204' 2024-08-31T22:24:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:29.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=120, tag_tid=14, entry_tid=22204' 2024-08-31T22:24:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=120, tag_tid=14, entry_tid=22204' 2024-08-31T22:24:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=22204 2024-08-31T22:24:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 22204 2024-08-31T22:24:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=121, tag_tid=14, entry_tid=22225], [object_number=120, tag_tid=14, entry_tid=22224], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=121, tag_tid=14, entry_tid=22225], [object_number=120, tag_tid=14, entry_tid=22224], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=121, tag_tid=14, entry_tid=22225' 2024-08-31T22:24:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=121, tag_tid=14, entry_tid=22225' 2024-08-31T22:24:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=121, tag_tid=14, entry_tid=22225' '!=' 'object_number=120, tag_tid=14, entry_tid=22204' 2024-08-31T22:24:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:24:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=121, tag_tid=14, entry_tid=22225' = 'object_number=120, tag_tid=14, entry_tid=22204' 2024-08-31T22:24:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=121, tag_tid=14, entry_tid=22225' 2024-08-31T22:24:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=15 2024-08-31T22:24:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=14, entry_tid=22225' 2024-08-31T22:24:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:24:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=14 2024-08-31T22:24:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:24:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=121, tag_tid=14, entry_tid=22225' 2024-08-31T22:24:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:24:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:24:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=22225 2024-08-31T22:24:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 15 = 14 -a 1 -le 22225 2024-08-31T22:24:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:24:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:24:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=121, tag_tid=14, entry_tid=22225], [object_number=120, tag_tid=14, entry_tid=22224], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=121, tag_tid=14, entry_tid=22225], [object_number=120, tag_tid=14, entry_tid=22224], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=121, tag_tid=14, entry_tid=22225' 2024-08-31T22:24:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=121, tag_tid=14, entry_tid=22225' 2024-08-31T22:24:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=121, tag_tid=14, entry_tid=22225' '!=' 'object_number=121, tag_tid=14, entry_tid=22225' 2024-08-31T22:24:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:24:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.4 2024-08-31T22:24:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:24:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:24:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:24:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:24:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:24:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:24:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:24:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:24:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:24:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:24:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:24:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:24:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:24:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:24:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:24:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:24:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:24:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:24:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:24:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:24:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:24:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:24:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:24:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:24:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:24:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:24:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:24:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:24:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:24:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:24:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:24:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:24:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:24:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:24:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:24:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:24:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:24:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:24:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:24:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:24:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:24:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:24:30.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:30.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:24:30.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:30.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:30.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:24:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:24:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:24:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:24:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:24:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:24:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:24:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:24:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:24:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:24:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:24:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:24:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 30 2024-08-31T22:24:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 30 2024-08-31T22:24:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:24:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=121, tag_tid=15, entry_tid=1], [object_number=120, tag_tid=15, entry_tid=0], [object_number=123, tag_tid=14, entry_tid=22223], [object_number=122, tag_tid=14, entry_tid=22222]]], state=connected] 2024-08-31T22:24:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:24:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=121, tag_tid=15, entry_tid=1' -a 'object_number=121, tag_tid=15, entry_tid=1' = 'object_number=121, tag_tid=15, entry_tid=1' 2024-08-31T22:24:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-08-31T22:24:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap5 2024-08-31T22:24:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:24:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:24:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:24:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-08-31T22:24:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'test': 2024-08-31T22:24:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:24:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:24:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 9 2024-08-31T22:24:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a87b4c6a1b 2024-08-31T22:24:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a87b4c6a1b 2024-08-31T22:24:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:24:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:24:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:24:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:24:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:14:57 2024 2024-08-31T22:24:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:22:15 2024 2024-08-31T22:24:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:24:06 2024 2024-08-31T22:24:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:24:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a87b4c6a1b 2024-08-31T22:24:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:24:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:24:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:24:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:24:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n snap4-clean ']' 2024-08-31T22:24:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap4-clean 2024-08-31T22:24:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap4-clean 2024-08-31T22:24:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:24:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:24:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:24:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:24:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:24:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:24:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:24:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:24:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:24:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:24:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap5 2024-08-31T22:24:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap5 2024-08-31T22:24:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:24:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:24:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:24:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:24:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:24:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:24:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:24:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:24:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:24:36.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:24:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ clean_snap_name=snap5-clean 2024-08-31T22:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-08-31T22:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap5-clean 2024-08-31T22:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-08-31T22:24:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:24:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror test 2024-08-31T22:24:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:24:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:24:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=5 2024-08-31T22:24:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:24:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-08-31T22:24:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 19851264 bytes 2024-08-31T22:24:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 85199872 bytes 2024-08-31T22:24:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:24:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:24:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:24:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-08-31T22:24:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:24:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:24:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:24:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:24:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:24:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-08-31T22:24:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:24:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:24:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:24:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:24:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:24:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-08-31T22:24:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='test: 2024-08-31T22:24:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: global_id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:24:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+replaying 2024-08-31T22:24:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: replaying, {"bytes_per_second":2927776.4,"entries_behind_primary":803,"entries_per_second":184.13,"non_primary_position":{"entry_tid":21423,"object_number":123,"tag_tid":14},"primary_position":{"entry_tid":1,"object_number":121,"tag_tid":15},"seconds_until_synced":4} 2024-08-31T22:24:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: service: mirror.0 on smithi032 2024-08-31T22:24:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:24:25 2024-08-31T22:24:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: peer_sites: 2024-08-31T22:24:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: name: cluster2 2024-08-31T22:24:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+stopped 2024-08-31T22:24:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: local image is primary 2024-08-31T22:24:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:24:40' 2024-08-31T22:24:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ state: .*up+replaying' 2024-08-31T22:24:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: state: up+replaying 2024-08-31T22:24:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ description: .*' 2024-08-31T22:24:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: description: replaying, {"bytes_per_second":2927776.4,"entries_behind_primary":803,"entries_per_second":184.13,"non_primary_position":{"entry_tid":21423,"object_number":123,"tag_tid":14},"primary_position":{"entry_tid":1,"object_number":121,"tag_tid":15},"seconds_until_synced":4} 2024-08-31T22:24:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:24:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:24:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^up+' 2024-08-31T22:24:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:24:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep 'service: *mirror..* on ' 2024-08-31T22:24:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: service: mirror.0 on smithi032 2024-08-31T22:24:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update 2024-08-31T22:24:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-08-31T22:24:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:24:25' 2024-08-31T22:24:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-08-31 22:24:25' 2024-08-31T22:24:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:24:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:24:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'prev_last_update=2024-08-31 22:24:25' 2024-08-31T22:24:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:24:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:24:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-08-31T22:24:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='OKOKOK142574264mirror.0smithi03219.1.1-244-g3091dfc0trueOK42614267mirror.1smithi03219.1.1-244-g3091dfc0falseOKtestdd2c705e-25cf-4565-981f-d48de48eee84up+replayingreplaying, {"bytes_per_second":2927776.4,"entries_behind_primary":803,"entries_per_second":184.13,"non_primary_position":{"entry_tid":21423,"object_number":123,"tag_tid":14},"primary_position":{"entry_tid":1,"object_number":121,"tag_tid":15},"seconds_until_synced":4}42574264mirror.0smithi0322024-08-31 22:24:25cluster24ca32b00-2164-486e-837e-e93795a16af1up+stoppedlocal image is primary2024-08-31 22:24:40' 2024-08-31T22:24:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update state 2024-08-31T22:24:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-08-31T22:24:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:24:25' 2024-08-31T22:24:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-08-31T22:24:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ state=up+replaying 2024-08-31T22:24:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:24:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep up+replaying 2024-08-31T22:24:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:24:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:24:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap6 2024-08-31T22:24:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap6 2024-08-31T22:24:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:24:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:24:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:24:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap6 2024-08-31T22:24:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:25:14.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-08-31T22:25:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:25:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-08-31T22:25:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:25:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:25:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:25:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:25:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:25:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror test started 2024-08-31T22:25:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:25:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:25:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:25:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:25:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-08-31T22:25:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:25:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:25:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:25:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:25:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:25:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:25:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:25:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:25:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:25:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:25:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:25:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-08-31T22:25:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:25:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:25:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:25:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:25:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:25:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:25:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:25:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:25:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:25:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:25:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:25:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:25:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:25:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:25:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:25:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:25:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:25:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:25:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:25:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:25:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:25:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:25:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:25:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:25:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:25:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:25:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:25:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:25:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:25:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:25:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-08-31T22:25:15.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:25:15.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:25:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:25:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:25:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:25:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:25:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:25:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:25:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:25:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:25:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:25:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:25:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:25:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:25:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:25:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:25:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:25:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:25:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:25:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:25:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:25:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:25:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:25:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:25:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:25:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:25:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:25:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:25:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:25:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:25:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:25:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:25:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:25:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:25:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:25:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:25:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:25:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:25:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:25:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:25:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:25:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:25:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:25:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:25:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:25:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:25:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:25:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:25:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:25:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:25:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:25:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:25:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:25:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:25:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:25:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:25:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:25:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:25:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:25:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:25:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:25:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:25:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:25:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:25:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:25:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:25:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:25:15.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:25:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:25:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:25:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:25:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:25:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:25:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:25:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:25:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:25:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 32 2024-08-31T22:25:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 32 2024-08-31T22:25:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:25:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=18, entry_tid=1], [object_number=128, tag_tid=18, entry_tid=0], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=129, tag_tid=17, entry_tid=5901], [object_number=128, tag_tid=17, entry_tid=5900], [object_number=131, tag_tid=17, entry_tid=5899], [object_number=130, tag_tid=17, entry_tid=5898]]], state=connected] 2024-08-31T22:25:15.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=129, tag_tid=18, entry_tid=1' 2024-08-31T22:25:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:25:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:25:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:25:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:25:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:25:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:25:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:25:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 32 2024-08-31T22:25:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 32 2024-08-31T22:25:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:25:15.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=18, entry_tid=1], [object_number=128, tag_tid=18, entry_tid=0], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:15.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=129, tag_tid=17, entry_tid=5901], [object_number=128, tag_tid=17, entry_tid=5900], [object_number=131, tag_tid=17, entry_tid=5899], [object_number=130, tag_tid=17, entry_tid=5898]]], state=connected] 2024-08-31T22:25:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=129, tag_tid=17, entry_tid=5901' 2024-08-31T22:25:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=129, tag_tid=18, entry_tid=1' -a 'object_number=129, tag_tid=18, entry_tid=1' = 'object_number=129, tag_tid=17, entry_tid=5901' 2024-08-31T22:25:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=129, tag_tid=17, entry_tid=5901' '!=' '' 2024-08-31T22:25:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:25:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=129, tag_tid=17, entry_tid=5901' = '' 2024-08-31T22:25:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=129, tag_tid=17, entry_tid=5901' 2024-08-31T22:25:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=129, tag_tid=18, entry_tid=1' 2024-08-31T22:25:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:25:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:25:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=18 2024-08-31T22:25:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=129, tag_tid=17, entry_tid=5901' 2024-08-31T22:25:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:25:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:25:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=17 2024-08-31T22:25:15.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=129, tag_tid=18, entry_tid=1' 2024-08-31T22:25:15.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:25:15.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:25:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:25:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=129, tag_tid=17, entry_tid=5901' 2024-08-31T22:25:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:25:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:25:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=5901 2024-08-31T22:25:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 18 = 17 -a 1 -le 5901 2024-08-31T22:25:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:25:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:25:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:25:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:25:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:25:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:25:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:25:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:25:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:25:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:25:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:25:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:25:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:25:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:25:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:25:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:25:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:25:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:25:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:25:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:25:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:25:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:25:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:25:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:25:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:25:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:25:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:25:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:25:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:25:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:25:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:25:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:25:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:25:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:25:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:25:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:25:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:25:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:25:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:25:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:25:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:25:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:25:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:25:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:25:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:25:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:25:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:25:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:25:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:25:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:25:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:25:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:25:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:25:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:25:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:25:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:25:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:25:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:25:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:25:16.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 32 2024-08-31T22:25:16.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 32 2024-08-31T22:25:16.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:25:16.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=18, entry_tid=1], [object_number=128, tag_tid=18, entry_tid=0], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:16.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=130, tag_tid=17, entry_tid=6002], [object_number=129, tag_tid=17, entry_tid=6001], [object_number=128, tag_tid=17, entry_tid=6000], [object_number=131, tag_tid=17, entry_tid=5999]]], state=connected] 2024-08-31T22:25:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=129, tag_tid=18, entry_tid=1' 2024-08-31T22:25:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:25:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:25:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:25:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:25:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:25:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:25:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:25:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 32 2024-08-31T22:25:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 32 2024-08-31T22:25:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:25:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=18, entry_tid=1], [object_number=128, tag_tid=18, entry_tid=0], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=130, tag_tid=17, entry_tid=6002], [object_number=129, tag_tid=17, entry_tid=6001], [object_number=128, tag_tid=17, entry_tid=6000], [object_number=131, tag_tid=17, entry_tid=5999]]], state=connected] 2024-08-31T22:25:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=130, tag_tid=17, entry_tid=6002' 2024-08-31T22:25:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=129, tag_tid=18, entry_tid=1' -a 'object_number=129, tag_tid=18, entry_tid=1' = 'object_number=130, tag_tid=17, entry_tid=6002' 2024-08-31T22:25:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=130, tag_tid=17, entry_tid=6002' '!=' 'object_number=129, tag_tid=17, entry_tid=5901' 2024-08-31T22:25:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:25:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=130, tag_tid=17, entry_tid=6002' = 'object_number=129, tag_tid=17, entry_tid=5901' 2024-08-31T22:25:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=130, tag_tid=17, entry_tid=6002' 2024-08-31T22:25:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=129, tag_tid=18, entry_tid=1' 2024-08-31T22:25:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:25:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:25:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=18 2024-08-31T22:25:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=130, tag_tid=17, entry_tid=6002' 2024-08-31T22:25:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:25:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:25:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=17 2024-08-31T22:25:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=129, tag_tid=18, entry_tid=1' 2024-08-31T22:25:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:25:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:25:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:25:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=130, tag_tid=17, entry_tid=6002' 2024-08-31T22:25:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:25:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:25:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6002 2024-08-31T22:25:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 18 = 17 -a 1 -le 6002 2024-08-31T22:25:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:25:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:25:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:25:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:25:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:25:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:25:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:25:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:25:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:25:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:25:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:25:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:25:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:25:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:25:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:25:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:25:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:25:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:25:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:25:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:25:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:25:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:25:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:25:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:25:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:25:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:25:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:25:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:25:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:25:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:25:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:25:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:25:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:25:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:25:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:25:16.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:25:16.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:25:16.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:25:16.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:25:16.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:25:16.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:25:16.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:25:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:25:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:25:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:25:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:25:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:25:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:25:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:25:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:25:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:25:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:25:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:25:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:25:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:25:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:25:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:25:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:25:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:25:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 32 2024-08-31T22:25:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 32 2024-08-31T22:25:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:25:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=18, entry_tid=1], [object_number=128, tag_tid=18, entry_tid=0], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=129, tag_tid=17, entry_tid=6101], [object_number=128, tag_tid=17, entry_tid=6100], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=129, tag_tid=18, entry_tid=1' 2024-08-31T22:25:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:25:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:25:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:25:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:25:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:25:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:25:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:25:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 32 2024-08-31T22:25:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 32 2024-08-31T22:25:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:25:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=18, entry_tid=1], [object_number=128, tag_tid=18, entry_tid=0], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=129, tag_tid=17, entry_tid=6101], [object_number=128, tag_tid=17, entry_tid=6100], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=129, tag_tid=17, entry_tid=6101' 2024-08-31T22:25:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=129, tag_tid=18, entry_tid=1' -a 'object_number=129, tag_tid=18, entry_tid=1' = 'object_number=129, tag_tid=17, entry_tid=6101' 2024-08-31T22:25:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=129, tag_tid=17, entry_tid=6101' '!=' 'object_number=130, tag_tid=17, entry_tid=6002' 2024-08-31T22:25:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:25:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=129, tag_tid=17, entry_tid=6101' = 'object_number=130, tag_tid=17, entry_tid=6002' 2024-08-31T22:25:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=129, tag_tid=17, entry_tid=6101' 2024-08-31T22:25:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=129, tag_tid=18, entry_tid=1' 2024-08-31T22:25:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:25:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:25:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=18 2024-08-31T22:25:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=129, tag_tid=17, entry_tid=6101' 2024-08-31T22:25:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:25:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:25:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=17 2024-08-31T22:25:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=129, tag_tid=18, entry_tid=1' 2024-08-31T22:25:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:25:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:25:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:25:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=129, tag_tid=17, entry_tid=6101' 2024-08-31T22:25:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:25:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:25:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6101 2024-08-31T22:25:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 18 = 17 -a 1 -le 6101 2024-08-31T22:25:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:25:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:25:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:25:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:25:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:25:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:25:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:25:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:25:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:25:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:25:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:25:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:25:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:25:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:25:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:25:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:25:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:25:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:25:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:25:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:25:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:25:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:25:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:25:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:25:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:25:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:25:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:25:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:25:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:25:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:25:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:25:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:25:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:25:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:25:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:25:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:25:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:25:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:25:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:25:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:25:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:25:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:25:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:25:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:25:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:25:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:25:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:25:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:25:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:25:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:25:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:25:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:25:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:25:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:25:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:25:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:25:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:25:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:25:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:25:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 32 2024-08-31T22:25:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 32 2024-08-31T22:25:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:25:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=18, entry_tid=1], [object_number=128, tag_tid=18, entry_tid=0], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=129, tag_tid=17, entry_tid=6101], [object_number=128, tag_tid=17, entry_tid=6100], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=129, tag_tid=18, entry_tid=1' 2024-08-31T22:25:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:25:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:25:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:25:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:25:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:25:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:25:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:25:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 32 2024-08-31T22:25:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 32 2024-08-31T22:25:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:25:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=18, entry_tid=1], [object_number=128, tag_tid=18, entry_tid=0], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=129, tag_tid=17, entry_tid=6101], [object_number=128, tag_tid=17, entry_tid=6100], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=129, tag_tid=17, entry_tid=6101' 2024-08-31T22:25:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=129, tag_tid=18, entry_tid=1' -a 'object_number=129, tag_tid=18, entry_tid=1' = 'object_number=129, tag_tid=17, entry_tid=6101' 2024-08-31T22:25:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=129, tag_tid=17, entry_tid=6101' '!=' 'object_number=129, tag_tid=17, entry_tid=6101' 2024-08-31T22:25:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:25:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.4 2024-08-31T22:25:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:25:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:25:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:25:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:25:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:25:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:25:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:25:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:25:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:25:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:25:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:25:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:25:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:25:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:25:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:25:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:25:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:25:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:25:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:25:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:25:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:25:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:25:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:25:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:25:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:25:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:25:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:25:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:25:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:25:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:25:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:25:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:25:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:25:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:25:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:25:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:25:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:25:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:25:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:25:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:25:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:25:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:25:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:25:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:25:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:25:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:25:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:25:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:25:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:25:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:25:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:25:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:25:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:25:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:25:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:25:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:25:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:18.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 32 2024-08-31T22:25:18.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 32 2024-08-31T22:25:18.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:25:18.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=18, entry_tid=1], [object_number=128, tag_tid=18, entry_tid=0], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:18.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=129, tag_tid=17, entry_tid=6101], [object_number=128, tag_tid=17, entry_tid=6100], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=129, tag_tid=18, entry_tid=1' 2024-08-31T22:25:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:25:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:25:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:25:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:25:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:25:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:25:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:25:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 32 2024-08-31T22:25:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 32 2024-08-31T22:25:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:25:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=18, entry_tid=1], [object_number=128, tag_tid=18, entry_tid=0], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=129, tag_tid=17, entry_tid=6101], [object_number=128, tag_tid=17, entry_tid=6100], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=129, tag_tid=17, entry_tid=6101' 2024-08-31T22:25:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=129, tag_tid=18, entry_tid=1' -a 'object_number=129, tag_tid=18, entry_tid=1' = 'object_number=129, tag_tid=17, entry_tid=6101' 2024-08-31T22:25:18.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=129, tag_tid=17, entry_tid=6101' '!=' 'object_number=129, tag_tid=17, entry_tid=6101' 2024-08-31T22:25:18.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:25:18.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.8 2024-08-31T22:25:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:25:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:25:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:25:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:25:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:25:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:25:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:25:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:25:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:25:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:25:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:25:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:25:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:25:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:25:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:25:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:25:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:25:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:25:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:25:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:25:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:25:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:25:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:25:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:25:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:25:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:25:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:25:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:25:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:25:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:25:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:25:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:25:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:25:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:25:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:25:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:25:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:25:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:25:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:25:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:25:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:25:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:25:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:25:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:25:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:25:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:25:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:25:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:25:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:25:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:25:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 32 2024-08-31T22:25:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 32 2024-08-31T22:25:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:25:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=18, entry_tid=1], [object_number=128, tag_tid=18, entry_tid=0], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=129, tag_tid=18, entry_tid=1], [object_number=128, tag_tid=18, entry_tid=0], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=129, tag_tid=18, entry_tid=1' 2024-08-31T22:25:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:25:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:25:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:25:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:25:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:25:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:25:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:25:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:25:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:25:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:25:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 32 2024-08-31T22:25:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 32 2024-08-31T22:25:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:25:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=18, entry_tid=1], [object_number=128, tag_tid=18, entry_tid=0], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=129, tag_tid=18, entry_tid=1], [object_number=128, tag_tid=18, entry_tid=0], [object_number=131, tag_tid=17, entry_tid=6099], [object_number=130, tag_tid=17, entry_tid=6098]]], state=connected] 2024-08-31T22:25:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:25:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=129, tag_tid=18, entry_tid=1' 2024-08-31T22:25:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=129, tag_tid=18, entry_tid=1' -a 'object_number=129, tag_tid=18, entry_tid=1' = 'object_number=129, tag_tid=18, entry_tid=1' 2024-08-31T22:25:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:25:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-08-31T22:25:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:25:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap6 2024-08-31T22:25:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:25:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:25:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:25:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-08-31T22:25:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'test': 2024-08-31T22:25:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:25:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:25:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 11 2024-08-31T22:25:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a87b4c6a1b 2024-08-31T22:25:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a87b4c6a1b 2024-08-31T22:25:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:25:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:25:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:25:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:25:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:14:57 2024 2024-08-31T22:25:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:24:32 2024 2024-08-31T22:25:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:25:07 2024 2024-08-31T22:25:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:25:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a87b4c6a1b 2024-08-31T22:25:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:25:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:25:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:25:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n snap5-clean ']' 2024-08-31T22:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap5-clean 2024-08-31T22:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap5-clean 2024-08-31T22:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:25:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:25:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:25:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:25:22.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:25:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:25:22.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:25:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:25:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap6 2024-08-31T22:25:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap6 2024-08-31T22:25:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:25:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:25:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:25:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:25:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:25:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:25:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:25:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:25:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:25:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:25:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:25:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ clean_snap_name=snap6-clean 2024-08-31T22:25:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-08-31T22:25:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:25:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap6-clean 2024-08-31T22:25:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-08-31T22:25:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:25:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror test 2024-08-31T22:25:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:25:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:25:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=14 2024-08-31T22:25:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:25:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-08-31T22:25:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 5639168 bytes 2024-08-31T22:25:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 22144000 bytes 2024-08-31T22:25:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 22764544 bytes 2024-08-31T22:25:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 38341632 bytes 2024-08-31T22:25:30.304 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:25:30.302+0000 7ff273ffe640 -1 reset not still connected to 0x56192f5356c0 2024-08-31T22:25:30.304 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:25:30.302+0000 7ff273ffe640 -1 reset not still connected to 0x5619304c52b0 2024-08-31T22:25:30.304 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:25:30.302+0000 7ff273ffe640 -1 reset not still connected to 0x5619309d0820 2024-08-31T22:25:30.304 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:25:30.302+0000 7ff273ffe640 -1 reset not still connected to 0x56193124c4e0 2024-08-31T22:25:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:500 IOs, 144502784 bytes 2024-08-31T22:25:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:600 IOs, 169827328 bytes 2024-08-31T22:25:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:700 IOs, 196262912 bytes 2024-08-31T22:25:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:800 IOs, 196893696 bytes 2024-08-31T22:25:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:900 IOs, 229711872 bytes 2024-08-31T22:25:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:25:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:25:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:25:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:25:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-08-31T22:25:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:25:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:25:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:25:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:25:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:25:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:25:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-08-31T22:25:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:25:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:25:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:25:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:25:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:25:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:25:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-08-31T22:25:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='test: 2024-08-31T22:25:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: global_id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:25:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+replaying 2024-08-31T22:25:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: replaying, {"bytes_per_second":2585861.17,"entries_behind_primary":0,"entries_per_second":159.4,"non_primary_position":{"entry_tid":1,"object_number":129,"tag_tid":18},"primary_position":{"entry_tid":1,"object_number":129,"tag_tid":18}} 2024-08-31T22:25:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: service: mirror.0 on smithi032 2024-08-31T22:25:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:25:25 2024-08-31T22:25:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: peer_sites: 2024-08-31T22:25:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: name: cluster2 2024-08-31T22:25:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+stopped 2024-08-31T22:25:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: local image is primary 2024-08-31T22:25:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:25:40' 2024-08-31T22:25:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ state: .*up+replaying' 2024-08-31T22:25:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: state: up+replaying 2024-08-31T22:25:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ description: .*' 2024-08-31T22:25:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: description: replaying, {"bytes_per_second":2585861.17,"entries_behind_primary":0,"entries_per_second":159.4,"non_primary_position":{"entry_tid":1,"object_number":129,"tag_tid":18},"primary_position":{"entry_tid":1,"object_number":129,"tag_tid":18}} 2024-08-31T22:25:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:25:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:25:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^up+' 2024-08-31T22:25:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:25:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep 'service: *mirror..* on ' 2024-08-31T22:25:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: service: mirror.0 on smithi032 2024-08-31T22:25:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update 2024-08-31T22:25:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-08-31T22:25:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:25:25' 2024-08-31T22:25:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-08-31 22:25:25' 2024-08-31T22:25:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:25:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:25:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'prev_last_update=2024-08-31 22:25:25' 2024-08-31T22:25:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:25:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:25:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-08-31T22:25:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='OKOKOK142574264mirror.0smithi03219.1.1-244-g3091dfc0trueOK42614267mirror.1smithi03219.1.1-244-g3091dfc0falseOKtestdd2c705e-25cf-4565-981f-d48de48eee84up+replayingreplaying, {"bytes_per_second":2585861.17,"entries_behind_primary":0,"entries_per_second":159.4,"non_primary_position":{"entry_tid":1,"object_number":129,"tag_tid":18},"primary_position":{"entry_tid":1,"object_number":129,"tag_tid":18}}42574264mirror.0smithi0322024-08-31 22:25:25cluster24ca32b00-2164-486e-837e-e93795a16af1up+stoppedlocal image is primary2024-08-31 22:25:40' 2024-08-31T22:25:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update state 2024-08-31T22:25:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-08-31T22:25:41.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:25:25' 2024-08-31T22:25:41.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-08-31T22:25:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ state=up+replaying 2024-08-31T22:25:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:25:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep up+replaying 2024-08-31T22:25:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:25:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:25:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:25:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap7 2024-08-31T22:25:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap7 2024-08-31T22:25:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:25:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:25:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:25:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap7 2024-08-31T22:25:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:26:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-08-31T22:26:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-08-31T22:26:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:26:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:26:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:26:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror test started 2024-08-31T22:26:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:26:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:26:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:26:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-08-31T22:26:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:26:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:26:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:26:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:26:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:26:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:26:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:26:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-08-31T22:26:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:26:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:26:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:26:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:26:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:26:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:26:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:26:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:26:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:26:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:26:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:26:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:26:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:26:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:26:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:26:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:26:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-08-31T22:26:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:26:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:26:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:26:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:26:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:26:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:26:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:26:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:26:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:26:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:26:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:26:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:26:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:26:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:26:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:26:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:26:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:13.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:13.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:13.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:13.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=7128], [object_number=139, tag_tid=20, entry_tid=7127], [object_number=138, tag_tid=20, entry_tid=7126], [object_number=137, tag_tid=20, entry_tid=7125]]], state=connected] 2024-08-31T22:26:13.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:13.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=7128], [object_number=139, tag_tid=20, entry_tid=7127], [object_number=138, tag_tid=20, entry_tid=7126], [object_number=137, tag_tid=20, entry_tid=7125]]], state=connected] 2024-08-31T22:26:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=136, tag_tid=20, entry_tid=7128' 2024-08-31T22:26:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=136, tag_tid=20, entry_tid=7128' 2024-08-31T22:26:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=7128' '!=' '' 2024-08-31T22:26:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=7128' = '' 2024-08-31T22:26:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=136, tag_tid=20, entry_tid=7128' 2024-08-31T22:26:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=7128' 2024-08-31T22:26:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=7128' 2024-08-31T22:26:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7128 2024-08-31T22:26:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 7128 2024-08-31T22:26:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=7220], [object_number=139, tag_tid=20, entry_tid=7219], [object_number=138, tag_tid=20, entry_tid=7218], [object_number=137, tag_tid=20, entry_tid=7217]]], state=connected] 2024-08-31T22:26:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=7220], [object_number=139, tag_tid=20, entry_tid=7219], [object_number=138, tag_tid=20, entry_tid=7218], [object_number=137, tag_tid=20, entry_tid=7217]]], state=connected] 2024-08-31T22:26:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=136, tag_tid=20, entry_tid=7220' 2024-08-31T22:26:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=136, tag_tid=20, entry_tid=7220' 2024-08-31T22:26:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=7220' '!=' 'object_number=136, tag_tid=20, entry_tid=7128' 2024-08-31T22:26:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=7220' = 'object_number=136, tag_tid=20, entry_tid=7128' 2024-08-31T22:26:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=136, tag_tid=20, entry_tid=7220' 2024-08-31T22:26:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=7220' 2024-08-31T22:26:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:14.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:14.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:14.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=7220' 2024-08-31T22:26:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7220 2024-08-31T22:26:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 7220 2024-08-31T22:26:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=7312], [object_number=139, tag_tid=20, entry_tid=7311], [object_number=138, tag_tid=20, entry_tid=7310], [object_number=137, tag_tid=20, entry_tid=7309]]], state=connected] 2024-08-31T22:26:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:14.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:14.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:14.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:14.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=7312], [object_number=139, tag_tid=20, entry_tid=7311], [object_number=138, tag_tid=20, entry_tid=7310], [object_number=137, tag_tid=20, entry_tid=7309]]], state=connected] 2024-08-31T22:26:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=136, tag_tid=20, entry_tid=7312' 2024-08-31T22:26:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=136, tag_tid=20, entry_tid=7312' 2024-08-31T22:26:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=7312' '!=' 'object_number=136, tag_tid=20, entry_tid=7220' 2024-08-31T22:26:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=7312' = 'object_number=136, tag_tid=20, entry_tid=7220' 2024-08-31T22:26:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=136, tag_tid=20, entry_tid=7312' 2024-08-31T22:26:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:14.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=7312' 2024-08-31T22:26:14.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:14.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:14.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=7312' 2024-08-31T22:26:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7312 2024-08-31T22:26:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 7312 2024-08-31T22:26:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=139, tag_tid=20, entry_tid=7391], [object_number=138, tag_tid=20, entry_tid=7390], [object_number=137, tag_tid=20, entry_tid=7389], [object_number=136, tag_tid=20, entry_tid=7388]]], state=connected] 2024-08-31T22:26:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=139, tag_tid=20, entry_tid=7391], [object_number=138, tag_tid=20, entry_tid=7390], [object_number=137, tag_tid=20, entry_tid=7389], [object_number=136, tag_tid=20, entry_tid=7388]]], state=connected] 2024-08-31T22:26:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=139, tag_tid=20, entry_tid=7391' 2024-08-31T22:26:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=139, tag_tid=20, entry_tid=7391' 2024-08-31T22:26:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=139, tag_tid=20, entry_tid=7391' '!=' 'object_number=136, tag_tid=20, entry_tid=7312' 2024-08-31T22:26:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=139, tag_tid=20, entry_tid=7391' = 'object_number=136, tag_tid=20, entry_tid=7312' 2024-08-31T22:26:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=139, tag_tid=20, entry_tid=7391' 2024-08-31T22:26:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=139, tag_tid=20, entry_tid=7391' 2024-08-31T22:26:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=139, tag_tid=20, entry_tid=7391' 2024-08-31T22:26:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7391 2024-08-31T22:26:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 7391 2024-08-31T22:26:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=7508], [object_number=139, tag_tid=20, entry_tid=7507], [object_number=138, tag_tid=20, entry_tid=7506], [object_number=137, tag_tid=20, entry_tid=7505]]], state=connected] 2024-08-31T22:26:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=7508], [object_number=139, tag_tid=20, entry_tid=7507], [object_number=138, tag_tid=20, entry_tid=7506], [object_number=137, tag_tid=20, entry_tid=7505]]], state=connected] 2024-08-31T22:26:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=136, tag_tid=20, entry_tid=7508' 2024-08-31T22:26:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=136, tag_tid=20, entry_tid=7508' 2024-08-31T22:26:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=7508' '!=' 'object_number=139, tag_tid=20, entry_tid=7391' 2024-08-31T22:26:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=7508' = 'object_number=139, tag_tid=20, entry_tid=7391' 2024-08-31T22:26:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=136, tag_tid=20, entry_tid=7508' 2024-08-31T22:26:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=7508' 2024-08-31T22:26:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=7508' 2024-08-31T22:26:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7508 2024-08-31T22:26:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 7508 2024-08-31T22:26:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=7596], [object_number=139, tag_tid=20, entry_tid=7595], [object_number=138, tag_tid=20, entry_tid=7594], [object_number=137, tag_tid=20, entry_tid=7593]]], state=connected] 2024-08-31T22:26:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=7596], [object_number=139, tag_tid=20, entry_tid=7595], [object_number=138, tag_tid=20, entry_tid=7594], [object_number=137, tag_tid=20, entry_tid=7593]]], state=connected] 2024-08-31T22:26:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=136, tag_tid=20, entry_tid=7596' 2024-08-31T22:26:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=136, tag_tid=20, entry_tid=7596' 2024-08-31T22:26:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=7596' '!=' 'object_number=136, tag_tid=20, entry_tid=7508' 2024-08-31T22:26:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=7596' = 'object_number=136, tag_tid=20, entry_tid=7508' 2024-08-31T22:26:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=136, tag_tid=20, entry_tid=7596' 2024-08-31T22:26:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:16.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=7596' 2024-08-31T22:26:16.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:16.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:16.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=7596' 2024-08-31T22:26:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7596 2024-08-31T22:26:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 7596 2024-08-31T22:26:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:16.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:16.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:16.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:16.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:16.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=7680], [object_number=139, tag_tid=20, entry_tid=7679], [object_number=138, tag_tid=20, entry_tid=7678], [object_number=137, tag_tid=20, entry_tid=7677]]], state=connected] 2024-08-31T22:26:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:16.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=7680], [object_number=139, tag_tid=20, entry_tid=7679], [object_number=138, tag_tid=20, entry_tid=7678], [object_number=137, tag_tid=20, entry_tid=7677]]], state=connected] 2024-08-31T22:26:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=136, tag_tid=20, entry_tid=7680' 2024-08-31T22:26:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=136, tag_tid=20, entry_tid=7680' 2024-08-31T22:26:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=7680' '!=' 'object_number=136, tag_tid=20, entry_tid=7596' 2024-08-31T22:26:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=7680' = 'object_number=136, tag_tid=20, entry_tid=7596' 2024-08-31T22:26:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=136, tag_tid=20, entry_tid=7680' 2024-08-31T22:26:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=7680' 2024-08-31T22:26:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=7680' 2024-08-31T22:26:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7680 2024-08-31T22:26:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 7680 2024-08-31T22:26:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=7800], [object_number=139, tag_tid=20, entry_tid=7799], [object_number=138, tag_tid=20, entry_tid=7798], [object_number=137, tag_tid=20, entry_tid=7797]]], state=connected] 2024-08-31T22:26:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=7800], [object_number=139, tag_tid=20, entry_tid=7799], [object_number=138, tag_tid=20, entry_tid=7798], [object_number=137, tag_tid=20, entry_tid=7797]]], state=connected] 2024-08-31T22:26:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=136, tag_tid=20, entry_tid=7800' 2024-08-31T22:26:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=136, tag_tid=20, entry_tid=7800' 2024-08-31T22:26:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=7800' '!=' 'object_number=136, tag_tid=20, entry_tid=7680' 2024-08-31T22:26:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=7800' = 'object_number=136, tag_tid=20, entry_tid=7680' 2024-08-31T22:26:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=136, tag_tid=20, entry_tid=7800' 2024-08-31T22:26:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=7800' 2024-08-31T22:26:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=7800' 2024-08-31T22:26:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7800 2024-08-31T22:26:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 7800 2024-08-31T22:26:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=7881], [object_number=136, tag_tid=20, entry_tid=7880], [object_number=139, tag_tid=20, entry_tid=7879], [object_number=138, tag_tid=20, entry_tid=7878]]], state=connected] 2024-08-31T22:26:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=7881], [object_number=136, tag_tid=20, entry_tid=7880], [object_number=139, tag_tid=20, entry_tid=7879], [object_number=138, tag_tid=20, entry_tid=7878]]], state=connected] 2024-08-31T22:26:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=137, tag_tid=20, entry_tid=7881' 2024-08-31T22:26:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=137, tag_tid=20, entry_tid=7881' 2024-08-31T22:26:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=7881' '!=' 'object_number=136, tag_tid=20, entry_tid=7800' 2024-08-31T22:26:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=7881' = 'object_number=136, tag_tid=20, entry_tid=7800' 2024-08-31T22:26:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=137, tag_tid=20, entry_tid=7881' 2024-08-31T22:26:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=7881' 2024-08-31T22:26:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=7881' 2024-08-31T22:26:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7881 2024-08-31T22:26:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 7881 2024-08-31T22:26:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:18.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:18.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:18.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:18.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=7960], [object_number=139, tag_tid=20, entry_tid=7959], [object_number=138, tag_tid=20, entry_tid=7958], [object_number=137, tag_tid=20, entry_tid=7957]]], state=connected] 2024-08-31T22:26:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:18.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:18.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:18.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:18.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=7960], [object_number=139, tag_tid=20, entry_tid=7959], [object_number=138, tag_tid=20, entry_tid=7958], [object_number=137, tag_tid=20, entry_tid=7957]]], state=connected] 2024-08-31T22:26:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=136, tag_tid=20, entry_tid=7960' 2024-08-31T22:26:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=136, tag_tid=20, entry_tid=7960' 2024-08-31T22:26:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=7960' '!=' 'object_number=137, tag_tid=20, entry_tid=7881' 2024-08-31T22:26:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=7960' = 'object_number=137, tag_tid=20, entry_tid=7881' 2024-08-31T22:26:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=136, tag_tid=20, entry_tid=7960' 2024-08-31T22:26:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=7960' 2024-08-31T22:26:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=7960' 2024-08-31T22:26:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7960 2024-08-31T22:26:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 7960 2024-08-31T22:26:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=8037], [object_number=136, tag_tid=20, entry_tid=8036], [object_number=139, tag_tid=20, entry_tid=8035], [object_number=138, tag_tid=20, entry_tid=8034]]], state=connected] 2024-08-31T22:26:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=8037], [object_number=136, tag_tid=20, entry_tid=8036], [object_number=139, tag_tid=20, entry_tid=8035], [object_number=138, tag_tid=20, entry_tid=8034]]], state=connected] 2024-08-31T22:26:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=137, tag_tid=20, entry_tid=8037' 2024-08-31T22:26:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=137, tag_tid=20, entry_tid=8037' 2024-08-31T22:26:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=8037' '!=' 'object_number=136, tag_tid=20, entry_tid=7960' 2024-08-31T22:26:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=8037' = 'object_number=136, tag_tid=20, entry_tid=7960' 2024-08-31T22:26:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=137, tag_tid=20, entry_tid=8037' 2024-08-31T22:26:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=8037' 2024-08-31T22:26:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=8037' 2024-08-31T22:26:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8037 2024-08-31T22:26:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 8037 2024-08-31T22:26:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:19.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:19.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:19.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:19.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:19.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:19.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:19.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:19.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:19.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=138, tag_tid=20, entry_tid=8114], [object_number=137, tag_tid=20, entry_tid=8113], [object_number=136, tag_tid=20, entry_tid=8112], [object_number=139, tag_tid=20, entry_tid=8111]]], state=connected] 2024-08-31T22:26:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:19.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:19.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:19.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:19.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:19.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:19.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=138, tag_tid=20, entry_tid=8114], [object_number=137, tag_tid=20, entry_tid=8113], [object_number=136, tag_tid=20, entry_tid=8112], [object_number=139, tag_tid=20, entry_tid=8111]]], state=connected] 2024-08-31T22:26:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=138, tag_tid=20, entry_tid=8114' 2024-08-31T22:26:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=138, tag_tid=20, entry_tid=8114' 2024-08-31T22:26:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=138, tag_tid=20, entry_tid=8114' '!=' 'object_number=137, tag_tid=20, entry_tid=8037' 2024-08-31T22:26:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=138, tag_tid=20, entry_tid=8114' = 'object_number=137, tag_tid=20, entry_tid=8037' 2024-08-31T22:26:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=138, tag_tid=20, entry_tid=8114' 2024-08-31T22:26:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=138, tag_tid=20, entry_tid=8114' 2024-08-31T22:26:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:19.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=138, tag_tid=20, entry_tid=8114' 2024-08-31T22:26:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8114 2024-08-31T22:26:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 8114 2024-08-31T22:26:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=8189], [object_number=136, tag_tid=20, entry_tid=8188], [object_number=139, tag_tid=20, entry_tid=8187], [object_number=138, tag_tid=20, entry_tid=8186]]], state=connected] 2024-08-31T22:26:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=8189], [object_number=136, tag_tid=20, entry_tid=8188], [object_number=139, tag_tid=20, entry_tid=8187], [object_number=138, tag_tid=20, entry_tid=8186]]], state=connected] 2024-08-31T22:26:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=137, tag_tid=20, entry_tid=8189' 2024-08-31T22:26:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=137, tag_tid=20, entry_tid=8189' 2024-08-31T22:26:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=8189' '!=' 'object_number=138, tag_tid=20, entry_tid=8114' 2024-08-31T22:26:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=8189' = 'object_number=138, tag_tid=20, entry_tid=8114' 2024-08-31T22:26:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=137, tag_tid=20, entry_tid=8189' 2024-08-31T22:26:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=8189' 2024-08-31T22:26:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=8189' 2024-08-31T22:26:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8189 2024-08-31T22:26:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 8189 2024-08-31T22:26:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:20.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:20.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:20.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:20.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:20.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:20.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:20.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=8264], [object_number=139, tag_tid=20, entry_tid=8263], [object_number=138, tag_tid=20, entry_tid=8262], [object_number=137, tag_tid=20, entry_tid=8261]]], state=connected] 2024-08-31T22:26:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=8264], [object_number=139, tag_tid=20, entry_tid=8263], [object_number=138, tag_tid=20, entry_tid=8262], [object_number=137, tag_tid=20, entry_tid=8261]]], state=connected] 2024-08-31T22:26:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=136, tag_tid=20, entry_tid=8264' 2024-08-31T22:26:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=136, tag_tid=20, entry_tid=8264' 2024-08-31T22:26:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=8264' '!=' 'object_number=137, tag_tid=20, entry_tid=8189' 2024-08-31T22:26:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=8264' = 'object_number=137, tag_tid=20, entry_tid=8189' 2024-08-31T22:26:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=136, tag_tid=20, entry_tid=8264' 2024-08-31T22:26:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=8264' 2024-08-31T22:26:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:20.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:20.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=8264' 2024-08-31T22:26:20.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:20.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8264 2024-08-31T22:26:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 8264 2024-08-31T22:26:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:20.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:20.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:20.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:20.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:20.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:20.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:20.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:20.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:20.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:21.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:21.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:21.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=8373], [object_number=136, tag_tid=20, entry_tid=8372], [object_number=139, tag_tid=20, entry_tid=8371], [object_number=138, tag_tid=20, entry_tid=8370]]], state=connected] 2024-08-31T22:26:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=8373], [object_number=136, tag_tid=20, entry_tid=8372], [object_number=139, tag_tid=20, entry_tid=8371], [object_number=138, tag_tid=20, entry_tid=8370]]], state=connected] 2024-08-31T22:26:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=137, tag_tid=20, entry_tid=8373' 2024-08-31T22:26:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=137, tag_tid=20, entry_tid=8373' 2024-08-31T22:26:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=8373' '!=' 'object_number=136, tag_tid=20, entry_tid=8264' 2024-08-31T22:26:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=8373' = 'object_number=136, tag_tid=20, entry_tid=8264' 2024-08-31T22:26:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=137, tag_tid=20, entry_tid=8373' 2024-08-31T22:26:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=8373' 2024-08-31T22:26:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=8373' 2024-08-31T22:26:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8373 2024-08-31T22:26:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 8373 2024-08-31T22:26:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=8449], [object_number=136, tag_tid=20, entry_tid=8448], [object_number=139, tag_tid=20, entry_tid=8447], [object_number=138, tag_tid=20, entry_tid=8446]]], state=connected] 2024-08-31T22:26:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=8449], [object_number=136, tag_tid=20, entry_tid=8448], [object_number=139, tag_tid=20, entry_tid=8447], [object_number=138, tag_tid=20, entry_tid=8446]]], state=connected] 2024-08-31T22:26:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=137, tag_tid=20, entry_tid=8449' 2024-08-31T22:26:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=137, tag_tid=20, entry_tid=8449' 2024-08-31T22:26:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=8449' '!=' 'object_number=137, tag_tid=20, entry_tid=8373' 2024-08-31T22:26:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=8449' = 'object_number=137, tag_tid=20, entry_tid=8373' 2024-08-31T22:26:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=137, tag_tid=20, entry_tid=8449' 2024-08-31T22:26:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=8449' 2024-08-31T22:26:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=8449' 2024-08-31T22:26:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8449 2024-08-31T22:26:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 8449 2024-08-31T22:26:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:21.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:22.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:22.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:22.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:22.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:22.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:22.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:22.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:22.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:22.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:22.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:22.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=8537], [object_number=136, tag_tid=20, entry_tid=8536], [object_number=139, tag_tid=20, entry_tid=8535], [object_number=138, tag_tid=20, entry_tid=8534]]], state=connected] 2024-08-31T22:26:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:22.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:22.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:22.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:22.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:22.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:22.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:22.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:22.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:22.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:22.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:22.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:22.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:22.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:22.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=8537], [object_number=136, tag_tid=20, entry_tid=8536], [object_number=139, tag_tid=20, entry_tid=8535], [object_number=138, tag_tid=20, entry_tid=8534]]], state=connected] 2024-08-31T22:26:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=137, tag_tid=20, entry_tid=8537' 2024-08-31T22:26:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=137, tag_tid=20, entry_tid=8537' 2024-08-31T22:26:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=8537' '!=' 'object_number=137, tag_tid=20, entry_tid=8449' 2024-08-31T22:26:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=8537' = 'object_number=137, tag_tid=20, entry_tid=8449' 2024-08-31T22:26:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=137, tag_tid=20, entry_tid=8537' 2024-08-31T22:26:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=8537' 2024-08-31T22:26:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=8537' 2024-08-31T22:26:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8537 2024-08-31T22:26:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 8537 2024-08-31T22:26:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:22.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:22.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:22.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:22.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:22.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=139, tag_tid=20, entry_tid=8635], [object_number=138, tag_tid=20, entry_tid=8634], [object_number=137, tag_tid=20, entry_tid=8633], [object_number=136, tag_tid=20, entry_tid=8632]]], state=connected] 2024-08-31T22:26:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=139, tag_tid=20, entry_tid=8635], [object_number=138, tag_tid=20, entry_tid=8634], [object_number=137, tag_tid=20, entry_tid=8633], [object_number=136, tag_tid=20, entry_tid=8632]]], state=connected] 2024-08-31T22:26:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=139, tag_tid=20, entry_tid=8635' 2024-08-31T22:26:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=139, tag_tid=20, entry_tid=8635' 2024-08-31T22:26:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=139, tag_tid=20, entry_tid=8635' '!=' 'object_number=137, tag_tid=20, entry_tid=8537' 2024-08-31T22:26:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=139, tag_tid=20, entry_tid=8635' = 'object_number=137, tag_tid=20, entry_tid=8537' 2024-08-31T22:26:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=139, tag_tid=20, entry_tid=8635' 2024-08-31T22:26:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=139, tag_tid=20, entry_tid=8635' 2024-08-31T22:26:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=139, tag_tid=20, entry_tid=8635' 2024-08-31T22:26:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8635 2024-08-31T22:26:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 8635 2024-08-31T22:26:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:23.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:23.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:23.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:23.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:23.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=8737], [object_number=136, tag_tid=20, entry_tid=8736], [object_number=139, tag_tid=20, entry_tid=8735], [object_number=138, tag_tid=20, entry_tid=8734]]], state=connected] 2024-08-31T22:26:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:23.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:23.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:23.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:23.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:23.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:23.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:23.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:23.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:23.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:23.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=8737], [object_number=136, tag_tid=20, entry_tid=8736], [object_number=139, tag_tid=20, entry_tid=8735], [object_number=138, tag_tid=20, entry_tid=8734]]], state=connected] 2024-08-31T22:26:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=137, tag_tid=20, entry_tid=8737' 2024-08-31T22:26:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=137, tag_tid=20, entry_tid=8737' 2024-08-31T22:26:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=8737' '!=' 'object_number=139, tag_tid=20, entry_tid=8635' 2024-08-31T22:26:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=8737' = 'object_number=139, tag_tid=20, entry_tid=8635' 2024-08-31T22:26:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=137, tag_tid=20, entry_tid=8737' 2024-08-31T22:26:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:23.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=8737' 2024-08-31T22:26:23.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:23.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=8737' 2024-08-31T22:26:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8737 2024-08-31T22:26:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 8737 2024-08-31T22:26:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:23.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:23.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:23.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=8837], [object_number=136, tag_tid=20, entry_tid=8836], [object_number=139, tag_tid=20, entry_tid=8835], [object_number=138, tag_tid=20, entry_tid=8834]]], state=connected] 2024-08-31T22:26:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=8837], [object_number=136, tag_tid=20, entry_tid=8836], [object_number=139, tag_tid=20, entry_tid=8835], [object_number=138, tag_tid=20, entry_tid=8834]]], state=connected] 2024-08-31T22:26:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=137, tag_tid=20, entry_tid=8837' 2024-08-31T22:26:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=137, tag_tid=20, entry_tid=8837' 2024-08-31T22:26:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=8837' '!=' 'object_number=137, tag_tid=20, entry_tid=8737' 2024-08-31T22:26:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=8837' = 'object_number=137, tag_tid=20, entry_tid=8737' 2024-08-31T22:26:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=137, tag_tid=20, entry_tid=8837' 2024-08-31T22:26:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=8837' 2024-08-31T22:26:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=8837' 2024-08-31T22:26:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8837 2024-08-31T22:26:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 8837 2024-08-31T22:26:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:24.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:24.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:24.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:24.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:24.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:24.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:24.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:24.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=8933], [object_number=136, tag_tid=20, entry_tid=8932], [object_number=139, tag_tid=20, entry_tid=8931], [object_number=138, tag_tid=20, entry_tid=8930]]], state=connected] 2024-08-31T22:26:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:24.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:24.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:24.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:24.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:24.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=8933], [object_number=136, tag_tid=20, entry_tid=8932], [object_number=139, tag_tid=20, entry_tid=8931], [object_number=138, tag_tid=20, entry_tid=8930]]], state=connected] 2024-08-31T22:26:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=137, tag_tid=20, entry_tid=8933' 2024-08-31T22:26:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=137, tag_tid=20, entry_tid=8933' 2024-08-31T22:26:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=8933' '!=' 'object_number=137, tag_tid=20, entry_tid=8837' 2024-08-31T22:26:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=8933' = 'object_number=137, tag_tid=20, entry_tid=8837' 2024-08-31T22:26:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=137, tag_tid=20, entry_tid=8933' 2024-08-31T22:26:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=8933' 2024-08-31T22:26:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=8933' 2024-08-31T22:26:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8933 2024-08-31T22:26:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 8933 2024-08-31T22:26:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=9065], [object_number=136, tag_tid=20, entry_tid=9064], [object_number=139, tag_tid=20, entry_tid=9063], [object_number=138, tag_tid=20, entry_tid=9062]]], state=connected] 2024-08-31T22:26:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=9065], [object_number=136, tag_tid=20, entry_tid=9064], [object_number=139, tag_tid=20, entry_tid=9063], [object_number=138, tag_tid=20, entry_tid=9062]]], state=connected] 2024-08-31T22:26:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=137, tag_tid=20, entry_tid=9065' 2024-08-31T22:26:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=137, tag_tid=20, entry_tid=9065' 2024-08-31T22:26:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=9065' '!=' 'object_number=137, tag_tid=20, entry_tid=8933' 2024-08-31T22:26:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=9065' = 'object_number=137, tag_tid=20, entry_tid=8933' 2024-08-31T22:26:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=137, tag_tid=20, entry_tid=9065' 2024-08-31T22:26:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=9065' 2024-08-31T22:26:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=9065' 2024-08-31T22:26:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9065 2024-08-31T22:26:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 9065 2024-08-31T22:26:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=139, tag_tid=20, entry_tid=9151], [object_number=138, tag_tid=20, entry_tid=9150], [object_number=137, tag_tid=20, entry_tid=9149], [object_number=136, tag_tid=20, entry_tid=9148]]], state=connected] 2024-08-31T22:26:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=139, tag_tid=20, entry_tid=9151], [object_number=138, tag_tid=20, entry_tid=9150], [object_number=137, tag_tid=20, entry_tid=9149], [object_number=136, tag_tid=20, entry_tid=9148]]], state=connected] 2024-08-31T22:26:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=139, tag_tid=20, entry_tid=9151' 2024-08-31T22:26:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=139, tag_tid=20, entry_tid=9151' 2024-08-31T22:26:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=139, tag_tid=20, entry_tid=9151' '!=' 'object_number=137, tag_tid=20, entry_tid=9065' 2024-08-31T22:26:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=139, tag_tid=20, entry_tid=9151' = 'object_number=137, tag_tid=20, entry_tid=9065' 2024-08-31T22:26:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=139, tag_tid=20, entry_tid=9151' 2024-08-31T22:26:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=139, tag_tid=20, entry_tid=9151' 2024-08-31T22:26:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=139, tag_tid=20, entry_tid=9151' 2024-08-31T22:26:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9151 2024-08-31T22:26:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 9151 2024-08-31T22:26:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:25.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:25.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:25.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:25.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:25.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:25.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=9237], [object_number=136, tag_tid=20, entry_tid=9236], [object_number=139, tag_tid=20, entry_tid=9235], [object_number=138, tag_tid=20, entry_tid=9234]]], state=connected] 2024-08-31T22:26:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=137, tag_tid=20, entry_tid=9237], [object_number=136, tag_tid=20, entry_tid=9236], [object_number=139, tag_tid=20, entry_tid=9235], [object_number=138, tag_tid=20, entry_tid=9234]]], state=connected] 2024-08-31T22:26:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=137, tag_tid=20, entry_tid=9237' 2024-08-31T22:26:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=137, tag_tid=20, entry_tid=9237' 2024-08-31T22:26:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=9237' '!=' 'object_number=139, tag_tid=20, entry_tid=9151' 2024-08-31T22:26:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=137, tag_tid=20, entry_tid=9237' = 'object_number=139, tag_tid=20, entry_tid=9151' 2024-08-31T22:26:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=137, tag_tid=20, entry_tid=9237' 2024-08-31T22:26:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=9237' 2024-08-31T22:26:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=137, tag_tid=20, entry_tid=9237' 2024-08-31T22:26:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9237 2024-08-31T22:26:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 9237 2024-08-31T22:26:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:26.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=138, tag_tid=20, entry_tid=9322], [object_number=137, tag_tid=20, entry_tid=9321], [object_number=136, tag_tid=20, entry_tid=9320], [object_number=139, tag_tid=20, entry_tid=9319]]], state=connected] 2024-08-31T22:26:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=138, tag_tid=20, entry_tid=9322], [object_number=137, tag_tid=20, entry_tid=9321], [object_number=136, tag_tid=20, entry_tid=9320], [object_number=139, tag_tid=20, entry_tid=9319]]], state=connected] 2024-08-31T22:26:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=138, tag_tid=20, entry_tid=9322' 2024-08-31T22:26:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=138, tag_tid=20, entry_tid=9322' 2024-08-31T22:26:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=138, tag_tid=20, entry_tid=9322' '!=' 'object_number=137, tag_tid=20, entry_tid=9237' 2024-08-31T22:26:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=138, tag_tid=20, entry_tid=9322' = 'object_number=137, tag_tid=20, entry_tid=9237' 2024-08-31T22:26:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=138, tag_tid=20, entry_tid=9322' 2024-08-31T22:26:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=138, tag_tid=20, entry_tid=9322' 2024-08-31T22:26:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=138, tag_tid=20, entry_tid=9322' 2024-08-31T22:26:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9322 2024-08-31T22:26:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 9322 2024-08-31T22:26:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:26.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:26.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:26.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:26.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:26.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:26.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:26.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:26.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:26.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=9404], [object_number=139, tag_tid=20, entry_tid=9403], [object_number=138, tag_tid=20, entry_tid=9402], [object_number=137, tag_tid=20, entry_tid=9401]]], state=connected] 2024-08-31T22:26:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=9404], [object_number=139, tag_tid=20, entry_tid=9403], [object_number=138, tag_tid=20, entry_tid=9402], [object_number=137, tag_tid=20, entry_tid=9401]]], state=connected] 2024-08-31T22:26:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=136, tag_tid=20, entry_tid=9404' 2024-08-31T22:26:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=136, tag_tid=20, entry_tid=9404' 2024-08-31T22:26:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=9404' '!=' 'object_number=138, tag_tid=20, entry_tid=9322' 2024-08-31T22:26:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=9404' = 'object_number=138, tag_tid=20, entry_tid=9322' 2024-08-31T22:26:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=136, tag_tid=20, entry_tid=9404' 2024-08-31T22:26:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=9404' 2024-08-31T22:26:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:26.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:26.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:26.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:26.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=9404' 2024-08-31T22:26:26.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:26.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9404 2024-08-31T22:26:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 9404 2024-08-31T22:26:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:27.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:27.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=139, tag_tid=20, entry_tid=9503], [object_number=138, tag_tid=20, entry_tid=9502], [object_number=137, tag_tid=20, entry_tid=9501], [object_number=136, tag_tid=20, entry_tid=9500]]], state=connected] 2024-08-31T22:26:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:27.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=139, tag_tid=20, entry_tid=9503], [object_number=138, tag_tid=20, entry_tid=9502], [object_number=137, tag_tid=20, entry_tid=9501], [object_number=136, tag_tid=20, entry_tid=9500]]], state=connected] 2024-08-31T22:26:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=139, tag_tid=20, entry_tid=9503' 2024-08-31T22:26:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=139, tag_tid=20, entry_tid=9503' 2024-08-31T22:26:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=139, tag_tid=20, entry_tid=9503' '!=' 'object_number=136, tag_tid=20, entry_tid=9404' 2024-08-31T22:26:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=139, tag_tid=20, entry_tid=9503' = 'object_number=136, tag_tid=20, entry_tid=9404' 2024-08-31T22:26:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=139, tag_tid=20, entry_tid=9503' 2024-08-31T22:26:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=139, tag_tid=20, entry_tid=9503' 2024-08-31T22:26:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=139, tag_tid=20, entry_tid=9503' 2024-08-31T22:26:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9503 2024-08-31T22:26:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 9503 2024-08-31T22:26:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:27.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:27.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:27.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:27.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:27.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:27.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=9604], [object_number=139, tag_tid=20, entry_tid=9603], [object_number=138, tag_tid=20, entry_tid=9602], [object_number=137, tag_tid=20, entry_tid=9601]]], state=connected] 2024-08-31T22:26:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:27.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:27.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:27.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=136, tag_tid=20, entry_tid=9604], [object_number=139, tag_tid=20, entry_tid=9603], [object_number=138, tag_tid=20, entry_tid=9602], [object_number=137, tag_tid=20, entry_tid=9601]]], state=connected] 2024-08-31T22:26:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=136, tag_tid=20, entry_tid=9604' 2024-08-31T22:26:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=136, tag_tid=20, entry_tid=9604' 2024-08-31T22:26:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=9604' '!=' 'object_number=139, tag_tid=20, entry_tid=9503' 2024-08-31T22:26:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=136, tag_tid=20, entry_tid=9604' = 'object_number=139, tag_tid=20, entry_tid=9503' 2024-08-31T22:26:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=136, tag_tid=20, entry_tid=9604' 2024-08-31T22:26:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:27.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=9604' 2024-08-31T22:26:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:27.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:27.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:27.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:27.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=136, tag_tid=20, entry_tid=9604' 2024-08-31T22:26:27.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:27.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9604 2024-08-31T22:26:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 9604 2024-08-31T22:26:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:28.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:28.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:28.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:28.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:28.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:28.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:28.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:28.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:28.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=139, tag_tid=20, entry_tid=9691], [object_number=138, tag_tid=20, entry_tid=9690], [object_number=137, tag_tid=20, entry_tid=9689], [object_number=136, tag_tid=20, entry_tid=9688]]], state=connected] 2024-08-31T22:26:28.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=139, tag_tid=20, entry_tid=9691], [object_number=138, tag_tid=20, entry_tid=9690], [object_number=137, tag_tid=20, entry_tid=9689], [object_number=136, tag_tid=20, entry_tid=9688]]], state=connected] 2024-08-31T22:26:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:28.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=139, tag_tid=20, entry_tid=9691' 2024-08-31T22:26:28.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=139, tag_tid=20, entry_tid=9691' 2024-08-31T22:26:28.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=139, tag_tid=20, entry_tid=9691' '!=' 'object_number=136, tag_tid=20, entry_tid=9604' 2024-08-31T22:26:28.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:28.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=139, tag_tid=20, entry_tid=9691' = 'object_number=136, tag_tid=20, entry_tid=9604' 2024-08-31T22:26:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=139, tag_tid=20, entry_tid=9691' 2024-08-31T22:26:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:28.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:28.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=139, tag_tid=20, entry_tid=9691' 2024-08-31T22:26:28.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:28.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:28.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:28.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:28.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:28.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=139, tag_tid=20, entry_tid=9691' 2024-08-31T22:26:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9691 2024-08-31T22:26:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 9691 2024-08-31T22:26:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:28.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:28.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:28.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=138, tag_tid=20, entry_tid=9754], [object_number=141, tag_tid=20, entry_tid=9753], [object_number=140, tag_tid=20, entry_tid=9752], [object_number=139, tag_tid=20, entry_tid=9751]]], state=connected] 2024-08-31T22:26:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=138, tag_tid=20, entry_tid=9754], [object_number=141, tag_tid=20, entry_tid=9753], [object_number=140, tag_tid=20, entry_tid=9752], [object_number=139, tag_tid=20, entry_tid=9751]]], state=connected] 2024-08-31T22:26:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:28.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=138, tag_tid=20, entry_tid=9754' 2024-08-31T22:26:28.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=138, tag_tid=20, entry_tid=9754' 2024-08-31T22:26:28.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=138, tag_tid=20, entry_tid=9754' '!=' 'object_number=139, tag_tid=20, entry_tid=9691' 2024-08-31T22:26:28.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:28.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=138, tag_tid=20, entry_tid=9754' = 'object_number=139, tag_tid=20, entry_tid=9691' 2024-08-31T22:26:28.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=138, tag_tid=20, entry_tid=9754' 2024-08-31T22:26:28.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:28.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:28.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=138, tag_tid=20, entry_tid=9754' 2024-08-31T22:26:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=138, tag_tid=20, entry_tid=9754' 2024-08-31T22:26:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9754 2024-08-31T22:26:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 9754 2024-08-31T22:26:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=9828], [object_number=143, tag_tid=20, entry_tid=9827], [object_number=138, tag_tid=20, entry_tid=9826], [object_number=141, tag_tid=20, entry_tid=9825]]], state=connected] 2024-08-31T22:26:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 34 2024-08-31T22:26:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=9828], [object_number=143, tag_tid=20, entry_tid=9827], [object_number=138, tag_tid=20, entry_tid=9826], [object_number=141, tag_tid=20, entry_tid=9825]]], state=connected] 2024-08-31T22:26:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=140, tag_tid=20, entry_tid=9828' 2024-08-31T22:26:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=140, tag_tid=20, entry_tid=9828' 2024-08-31T22:26:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=9828' '!=' 'object_number=138, tag_tid=20, entry_tid=9754' 2024-08-31T22:26:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=9828' = 'object_number=138, tag_tid=20, entry_tid=9754' 2024-08-31T22:26:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=140, tag_tid=20, entry_tid=9828' 2024-08-31T22:26:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=9828' 2024-08-31T22:26:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:29.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:29.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:29.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:29.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=9828' 2024-08-31T22:26:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9828 2024-08-31T22:26:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 9828 2024-08-31T22:26:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:29.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=9896], [object_number=143, tag_tid=20, entry_tid=9895], [object_number=142, tag_tid=20, entry_tid=9894], [object_number=141, tag_tid=20, entry_tid=9893]]], state=connected] 2024-08-31T22:26:29.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=9896], [object_number=143, tag_tid=20, entry_tid=9895], [object_number=142, tag_tid=20, entry_tid=9894], [object_number=141, tag_tid=20, entry_tid=9893]]], state=connected] 2024-08-31T22:26:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=140, tag_tid=20, entry_tid=9896' 2024-08-31T22:26:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=140, tag_tid=20, entry_tid=9896' 2024-08-31T22:26:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=9896' '!=' 'object_number=140, tag_tid=20, entry_tid=9828' 2024-08-31T22:26:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=9896' = 'object_number=140, tag_tid=20, entry_tid=9828' 2024-08-31T22:26:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=140, tag_tid=20, entry_tid=9896' 2024-08-31T22:26:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=9896' 2024-08-31T22:26:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:29.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:29.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:29.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:29.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=9896' 2024-08-31T22:26:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9896 2024-08-31T22:26:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 9896 2024-08-31T22:26:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:30.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:30.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:30.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:30.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:30.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=9979], [object_number=142, tag_tid=20, entry_tid=9978], [object_number=141, tag_tid=20, entry_tid=9977], [object_number=140, tag_tid=20, entry_tid=9976]]], state=connected] 2024-08-31T22:26:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=9979], [object_number=142, tag_tid=20, entry_tid=9978], [object_number=141, tag_tid=20, entry_tid=9977], [object_number=140, tag_tid=20, entry_tid=9976]]], state=connected] 2024-08-31T22:26:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=143, tag_tid=20, entry_tid=9979' 2024-08-31T22:26:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=143, tag_tid=20, entry_tid=9979' 2024-08-31T22:26:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=9979' '!=' 'object_number=140, tag_tid=20, entry_tid=9896' 2024-08-31T22:26:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=9979' = 'object_number=140, tag_tid=20, entry_tid=9896' 2024-08-31T22:26:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=143, tag_tid=20, entry_tid=9979' 2024-08-31T22:26:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=9979' 2024-08-31T22:26:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:30.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=9979' 2024-08-31T22:26:30.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:30.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9979 2024-08-31T22:26:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 9979 2024-08-31T22:26:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:30.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:30.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:30.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:30.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:30.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=141, tag_tid=20, entry_tid=10065], [object_number=140, tag_tid=20, entry_tid=10064], [object_number=143, tag_tid=20, entry_tid=10063], [object_number=142, tag_tid=20, entry_tid=10062]]], state=connected] 2024-08-31T22:26:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=141, tag_tid=20, entry_tid=10065], [object_number=140, tag_tid=20, entry_tid=10064], [object_number=143, tag_tid=20, entry_tid=10063], [object_number=142, tag_tid=20, entry_tid=10062]]], state=connected] 2024-08-31T22:26:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=141, tag_tid=20, entry_tid=10065' 2024-08-31T22:26:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=141, tag_tid=20, entry_tid=10065' 2024-08-31T22:26:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=141, tag_tid=20, entry_tid=10065' '!=' 'object_number=143, tag_tid=20, entry_tid=9979' 2024-08-31T22:26:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=141, tag_tid=20, entry_tid=10065' = 'object_number=143, tag_tid=20, entry_tid=9979' 2024-08-31T22:26:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=141, tag_tid=20, entry_tid=10065' 2024-08-31T22:26:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=141, tag_tid=20, entry_tid=10065' 2024-08-31T22:26:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=141, tag_tid=20, entry_tid=10065' 2024-08-31T22:26:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10065 2024-08-31T22:26:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 10065 2024-08-31T22:26:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:31.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:31.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=10148], [object_number=143, tag_tid=20, entry_tid=10147], [object_number=142, tag_tid=20, entry_tid=10146], [object_number=141, tag_tid=20, entry_tid=10145]]], state=connected] 2024-08-31T22:26:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:31.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=10148], [object_number=143, tag_tid=20, entry_tid=10147], [object_number=142, tag_tid=20, entry_tid=10146], [object_number=141, tag_tid=20, entry_tid=10145]]], state=connected] 2024-08-31T22:26:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=140, tag_tid=20, entry_tid=10148' 2024-08-31T22:26:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=140, tag_tid=20, entry_tid=10148' 2024-08-31T22:26:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=10148' '!=' 'object_number=141, tag_tid=20, entry_tid=10065' 2024-08-31T22:26:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=10148' = 'object_number=141, tag_tid=20, entry_tid=10065' 2024-08-31T22:26:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=140, tag_tid=20, entry_tid=10148' 2024-08-31T22:26:31.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:31.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:31.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=10148' 2024-08-31T22:26:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=10148' 2024-08-31T22:26:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10148 2024-08-31T22:26:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 10148 2024-08-31T22:26:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=10272], [object_number=143, tag_tid=20, entry_tid=10271], [object_number=142, tag_tid=20, entry_tid=10270], [object_number=141, tag_tid=20, entry_tid=10269]]], state=connected] 2024-08-31T22:26:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=10272], [object_number=143, tag_tid=20, entry_tid=10271], [object_number=142, tag_tid=20, entry_tid=10270], [object_number=141, tag_tid=20, entry_tid=10269]]], state=connected] 2024-08-31T22:26:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=140, tag_tid=20, entry_tid=10272' 2024-08-31T22:26:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=140, tag_tid=20, entry_tid=10272' 2024-08-31T22:26:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=10272' '!=' 'object_number=140, tag_tid=20, entry_tid=10148' 2024-08-31T22:26:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=10272' = 'object_number=140, tag_tid=20, entry_tid=10148' 2024-08-31T22:26:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=140, tag_tid=20, entry_tid=10272' 2024-08-31T22:26:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=10272' 2024-08-31T22:26:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=10272' 2024-08-31T22:26:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10272 2024-08-31T22:26:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 10272 2024-08-31T22:26:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=10370], [object_number=141, tag_tid=20, entry_tid=10369], [object_number=140, tag_tid=20, entry_tid=10368], [object_number=143, tag_tid=20, entry_tid=10367]]], state=connected] 2024-08-31T22:26:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=10370], [object_number=141, tag_tid=20, entry_tid=10369], [object_number=140, tag_tid=20, entry_tid=10368], [object_number=143, tag_tid=20, entry_tid=10367]]], state=connected] 2024-08-31T22:26:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=142, tag_tid=20, entry_tid=10370' 2024-08-31T22:26:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=142, tag_tid=20, entry_tid=10370' 2024-08-31T22:26:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=10370' '!=' 'object_number=140, tag_tid=20, entry_tid=10272' 2024-08-31T22:26:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=10370' = 'object_number=140, tag_tid=20, entry_tid=10272' 2024-08-31T22:26:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=142, tag_tid=20, entry_tid=10370' 2024-08-31T22:26:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=10370' 2024-08-31T22:26:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:32.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=10370' 2024-08-31T22:26:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10370 2024-08-31T22:26:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 10370 2024-08-31T22:26:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=10463], [object_number=142, tag_tid=20, entry_tid=10462], [object_number=141, tag_tid=20, entry_tid=10461], [object_number=140, tag_tid=20, entry_tid=10460]]], state=connected] 2024-08-31T22:26:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:33.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:33.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:33.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=10463], [object_number=142, tag_tid=20, entry_tid=10462], [object_number=141, tag_tid=20, entry_tid=10461], [object_number=140, tag_tid=20, entry_tid=10460]]], state=connected] 2024-08-31T22:26:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=143, tag_tid=20, entry_tid=10463' 2024-08-31T22:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=143, tag_tid=20, entry_tid=10463' 2024-08-31T22:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=10463' '!=' 'object_number=142, tag_tid=20, entry_tid=10370' 2024-08-31T22:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=10463' = 'object_number=142, tag_tid=20, entry_tid=10370' 2024-08-31T22:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=143, tag_tid=20, entry_tid=10463' 2024-08-31T22:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=10463' 2024-08-31T22:26:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=10463' 2024-08-31T22:26:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10463 2024-08-31T22:26:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 10463 2024-08-31T22:26:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:33.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:33.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:33.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:33.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:33.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:33.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:33.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=10552], [object_number=143, tag_tid=20, entry_tid=10551], [object_number=142, tag_tid=20, entry_tid=10550], [object_number=141, tag_tid=20, entry_tid=10549]]], state=connected] 2024-08-31T22:26:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=10552], [object_number=143, tag_tid=20, entry_tid=10551], [object_number=142, tag_tid=20, entry_tid=10550], [object_number=141, tag_tid=20, entry_tid=10549]]], state=connected] 2024-08-31T22:26:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=140, tag_tid=20, entry_tid=10552' 2024-08-31T22:26:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=140, tag_tid=20, entry_tid=10552' 2024-08-31T22:26:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=10552' '!=' 'object_number=143, tag_tid=20, entry_tid=10463' 2024-08-31T22:26:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=10552' = 'object_number=143, tag_tid=20, entry_tid=10463' 2024-08-31T22:26:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=140, tag_tid=20, entry_tid=10552' 2024-08-31T22:26:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=10552' 2024-08-31T22:26:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:33.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:33.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:33.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:33.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=10552' 2024-08-31T22:26:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10552 2024-08-31T22:26:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 10552 2024-08-31T22:26:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:33.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:33.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:33.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:33.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:33.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:33.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:33.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=141, tag_tid=20, entry_tid=10641], [object_number=140, tag_tid=20, entry_tid=10640], [object_number=143, tag_tid=20, entry_tid=10639], [object_number=142, tag_tid=20, entry_tid=10638]]], state=connected] 2024-08-31T22:26:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=141, tag_tid=20, entry_tid=10641], [object_number=140, tag_tid=20, entry_tid=10640], [object_number=143, tag_tid=20, entry_tid=10639], [object_number=142, tag_tid=20, entry_tid=10638]]], state=connected] 2024-08-31T22:26:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=141, tag_tid=20, entry_tid=10641' 2024-08-31T22:26:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=141, tag_tid=20, entry_tid=10641' 2024-08-31T22:26:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=141, tag_tid=20, entry_tid=10641' '!=' 'object_number=140, tag_tid=20, entry_tid=10552' 2024-08-31T22:26:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=141, tag_tid=20, entry_tid=10641' = 'object_number=140, tag_tid=20, entry_tid=10552' 2024-08-31T22:26:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=141, tag_tid=20, entry_tid=10641' 2024-08-31T22:26:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:34.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=141, tag_tid=20, entry_tid=10641' 2024-08-31T22:26:34.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:34.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=141, tag_tid=20, entry_tid=10641' 2024-08-31T22:26:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10641 2024-08-31T22:26:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 10641 2024-08-31T22:26:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=141, tag_tid=20, entry_tid=10729], [object_number=140, tag_tid=20, entry_tid=10728], [object_number=143, tag_tid=20, entry_tid=10727], [object_number=142, tag_tid=20, entry_tid=10726]]], state=connected] 2024-08-31T22:26:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=141, tag_tid=20, entry_tid=10729], [object_number=140, tag_tid=20, entry_tid=10728], [object_number=143, tag_tid=20, entry_tid=10727], [object_number=142, tag_tid=20, entry_tid=10726]]], state=connected] 2024-08-31T22:26:34.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=141, tag_tid=20, entry_tid=10729' 2024-08-31T22:26:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=141, tag_tid=20, entry_tid=10729' 2024-08-31T22:26:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=141, tag_tid=20, entry_tid=10729' '!=' 'object_number=141, tag_tid=20, entry_tid=10641' 2024-08-31T22:26:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=141, tag_tid=20, entry_tid=10729' = 'object_number=141, tag_tid=20, entry_tid=10641' 2024-08-31T22:26:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=141, tag_tid=20, entry_tid=10729' 2024-08-31T22:26:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=141, tag_tid=20, entry_tid=10729' 2024-08-31T22:26:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=141, tag_tid=20, entry_tid=10729' 2024-08-31T22:26:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10729 2024-08-31T22:26:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 10729 2024-08-31T22:26:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:34.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:34.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:34.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:34.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:34.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:34.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:34.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:34.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=10814], [object_number=141, tag_tid=20, entry_tid=10813], [object_number=140, tag_tid=20, entry_tid=10812], [object_number=143, tag_tid=20, entry_tid=10811]]], state=connected] 2024-08-31T22:26:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=10814], [object_number=141, tag_tid=20, entry_tid=10813], [object_number=140, tag_tid=20, entry_tid=10812], [object_number=143, tag_tid=20, entry_tid=10811]]], state=connected] 2024-08-31T22:26:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=142, tag_tid=20, entry_tid=10814' 2024-08-31T22:26:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=142, tag_tid=20, entry_tid=10814' 2024-08-31T22:26:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=10814' '!=' 'object_number=141, tag_tid=20, entry_tid=10729' 2024-08-31T22:26:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=10814' = 'object_number=141, tag_tid=20, entry_tid=10729' 2024-08-31T22:26:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=142, tag_tid=20, entry_tid=10814' 2024-08-31T22:26:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=10814' 2024-08-31T22:26:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=10814' 2024-08-31T22:26:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10814 2024-08-31T22:26:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 10814 2024-08-31T22:26:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=10923], [object_number=142, tag_tid=20, entry_tid=10922], [object_number=141, tag_tid=20, entry_tid=10921], [object_number=140, tag_tid=20, entry_tid=10920]]], state=connected] 2024-08-31T22:26:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=10923], [object_number=142, tag_tid=20, entry_tid=10922], [object_number=141, tag_tid=20, entry_tid=10921], [object_number=140, tag_tid=20, entry_tid=10920]]], state=connected] 2024-08-31T22:26:35.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=143, tag_tid=20, entry_tid=10923' 2024-08-31T22:26:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=143, tag_tid=20, entry_tid=10923' 2024-08-31T22:26:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=10923' '!=' 'object_number=142, tag_tid=20, entry_tid=10814' 2024-08-31T22:26:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=10923' = 'object_number=142, tag_tid=20, entry_tid=10814' 2024-08-31T22:26:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=143, tag_tid=20, entry_tid=10923' 2024-08-31T22:26:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=10923' 2024-08-31T22:26:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=10923' 2024-08-31T22:26:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10923 2024-08-31T22:26:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 10923 2024-08-31T22:26:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=11016], [object_number=143, tag_tid=20, entry_tid=11015], [object_number=142, tag_tid=20, entry_tid=11014], [object_number=141, tag_tid=20, entry_tid=11013]]], state=connected] 2024-08-31T22:26:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=11016], [object_number=143, tag_tid=20, entry_tid=11015], [object_number=142, tag_tid=20, entry_tid=11014], [object_number=141, tag_tid=20, entry_tid=11013]]], state=connected] 2024-08-31T22:26:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=140, tag_tid=20, entry_tid=11016' 2024-08-31T22:26:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=140, tag_tid=20, entry_tid=11016' 2024-08-31T22:26:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=11016' '!=' 'object_number=143, tag_tid=20, entry_tid=10923' 2024-08-31T22:26:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=11016' = 'object_number=143, tag_tid=20, entry_tid=10923' 2024-08-31T22:26:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=140, tag_tid=20, entry_tid=11016' 2024-08-31T22:26:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=11016' 2024-08-31T22:26:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=11016' 2024-08-31T22:26:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11016 2024-08-31T22:26:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 11016 2024-08-31T22:26:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=11102], [object_number=141, tag_tid=20, entry_tid=11101], [object_number=140, tag_tid=20, entry_tid=11100], [object_number=143, tag_tid=20, entry_tid=11099]]], state=connected] 2024-08-31T22:26:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=11102], [object_number=141, tag_tid=20, entry_tid=11101], [object_number=140, tag_tid=20, entry_tid=11100], [object_number=143, tag_tid=20, entry_tid=11099]]], state=connected] 2024-08-31T22:26:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=142, tag_tid=20, entry_tid=11102' 2024-08-31T22:26:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=142, tag_tid=20, entry_tid=11102' 2024-08-31T22:26:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=11102' '!=' 'object_number=140, tag_tid=20, entry_tid=11016' 2024-08-31T22:26:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=11102' = 'object_number=140, tag_tid=20, entry_tid=11016' 2024-08-31T22:26:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=142, tag_tid=20, entry_tid=11102' 2024-08-31T22:26:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=11102' 2024-08-31T22:26:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=11102' 2024-08-31T22:26:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11102 2024-08-31T22:26:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 11102 2024-08-31T22:26:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=11192], [object_number=143, tag_tid=20, entry_tid=11191], [object_number=142, tag_tid=20, entry_tid=11190], [object_number=141, tag_tid=20, entry_tid=11189]]], state=connected] 2024-08-31T22:26:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=11192], [object_number=143, tag_tid=20, entry_tid=11191], [object_number=142, tag_tid=20, entry_tid=11190], [object_number=141, tag_tid=20, entry_tid=11189]]], state=connected] 2024-08-31T22:26:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=140, tag_tid=20, entry_tid=11192' 2024-08-31T22:26:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=140, tag_tid=20, entry_tid=11192' 2024-08-31T22:26:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=11192' '!=' 'object_number=142, tag_tid=20, entry_tid=11102' 2024-08-31T22:26:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=11192' = 'object_number=142, tag_tid=20, entry_tid=11102' 2024-08-31T22:26:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=140, tag_tid=20, entry_tid=11192' 2024-08-31T22:26:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=11192' 2024-08-31T22:26:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=11192' 2024-08-31T22:26:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11192 2024-08-31T22:26:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 11192 2024-08-31T22:26:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:37.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:37.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=141, tag_tid=20, entry_tid=11285], [object_number=140, tag_tid=20, entry_tid=11284], [object_number=143, tag_tid=20, entry_tid=11283], [object_number=142, tag_tid=20, entry_tid=11282]]], state=connected] 2024-08-31T22:26:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=141, tag_tid=20, entry_tid=11285], [object_number=140, tag_tid=20, entry_tid=11284], [object_number=143, tag_tid=20, entry_tid=11283], [object_number=142, tag_tid=20, entry_tid=11282]]], state=connected] 2024-08-31T22:26:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=141, tag_tid=20, entry_tid=11285' 2024-08-31T22:26:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=141, tag_tid=20, entry_tid=11285' 2024-08-31T22:26:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=141, tag_tid=20, entry_tid=11285' '!=' 'object_number=140, tag_tid=20, entry_tid=11192' 2024-08-31T22:26:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=141, tag_tid=20, entry_tid=11285' = 'object_number=140, tag_tid=20, entry_tid=11192' 2024-08-31T22:26:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=141, tag_tid=20, entry_tid=11285' 2024-08-31T22:26:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=141, tag_tid=20, entry_tid=11285' 2024-08-31T22:26:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=141, tag_tid=20, entry_tid=11285' 2024-08-31T22:26:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11285 2024-08-31T22:26:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 11285 2024-08-31T22:26:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:37.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=11378], [object_number=141, tag_tid=20, entry_tid=11377], [object_number=140, tag_tid=20, entry_tid=11376], [object_number=143, tag_tid=20, entry_tid=11375]]], state=connected] 2024-08-31T22:26:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=11378], [object_number=141, tag_tid=20, entry_tid=11377], [object_number=140, tag_tid=20, entry_tid=11376], [object_number=143, tag_tid=20, entry_tid=11375]]], state=connected] 2024-08-31T22:26:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=142, tag_tid=20, entry_tid=11378' 2024-08-31T22:26:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=142, tag_tid=20, entry_tid=11378' 2024-08-31T22:26:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=11378' '!=' 'object_number=141, tag_tid=20, entry_tid=11285' 2024-08-31T22:26:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=11378' = 'object_number=141, tag_tid=20, entry_tid=11285' 2024-08-31T22:26:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=142, tag_tid=20, entry_tid=11378' 2024-08-31T22:26:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=11378' 2024-08-31T22:26:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=11378' 2024-08-31T22:26:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11378 2024-08-31T22:26:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 11378 2024-08-31T22:26:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:38.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:38.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:38.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:38.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:38.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:38.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:38.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:38.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:38.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:38.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:38.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:38.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:38.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:38.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:38.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:38.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=11466], [object_number=141, tag_tid=20, entry_tid=11465], [object_number=140, tag_tid=20, entry_tid=11464], [object_number=143, tag_tid=20, entry_tid=11463]]], state=connected] 2024-08-31T22:26:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=11466], [object_number=141, tag_tid=20, entry_tid=11465], [object_number=140, tag_tid=20, entry_tid=11464], [object_number=143, tag_tid=20, entry_tid=11463]]], state=connected] 2024-08-31T22:26:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=142, tag_tid=20, entry_tid=11466' 2024-08-31T22:26:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=142, tag_tid=20, entry_tid=11466' 2024-08-31T22:26:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=11466' '!=' 'object_number=142, tag_tid=20, entry_tid=11378' 2024-08-31T22:26:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=11466' = 'object_number=142, tag_tid=20, entry_tid=11378' 2024-08-31T22:26:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=142, tag_tid=20, entry_tid=11466' 2024-08-31T22:26:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=11466' 2024-08-31T22:26:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=11466' 2024-08-31T22:26:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11466 2024-08-31T22:26:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 11466 2024-08-31T22:26:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:38.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=11558], [object_number=141, tag_tid=20, entry_tid=11557], [object_number=140, tag_tid=20, entry_tid=11556], [object_number=143, tag_tid=20, entry_tid=11555]]], state=connected] 2024-08-31T22:26:38.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=11558], [object_number=141, tag_tid=20, entry_tid=11557], [object_number=140, tag_tid=20, entry_tid=11556], [object_number=143, tag_tid=20, entry_tid=11555]]], state=connected] 2024-08-31T22:26:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=142, tag_tid=20, entry_tid=11558' 2024-08-31T22:26:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=142, tag_tid=20, entry_tid=11558' 2024-08-31T22:26:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=11558' '!=' 'object_number=142, tag_tid=20, entry_tid=11466' 2024-08-31T22:26:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=11558' = 'object_number=142, tag_tid=20, entry_tid=11466' 2024-08-31T22:26:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=142, tag_tid=20, entry_tid=11558' 2024-08-31T22:26:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=11558' 2024-08-31T22:26:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:39.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:39.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:39.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:39.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:39.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=11558' 2024-08-31T22:26:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11558 2024-08-31T22:26:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 11558 2024-08-31T22:26:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:39.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:39.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:39.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=11654], [object_number=141, tag_tid=20, entry_tid=11653], [object_number=140, tag_tid=20, entry_tid=11652], [object_number=143, tag_tid=20, entry_tid=11651]]], state=connected] 2024-08-31T22:26:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=11675], [object_number=142, tag_tid=20, entry_tid=11674], [object_number=141, tag_tid=20, entry_tid=11673], [object_number=140, tag_tid=20, entry_tid=11672]]], state=connected] 2024-08-31T22:26:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=143, tag_tid=20, entry_tid=11675' 2024-08-31T22:26:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=143, tag_tid=20, entry_tid=11675' 2024-08-31T22:26:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=11675' '!=' 'object_number=142, tag_tid=20, entry_tid=11558' 2024-08-31T22:26:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=11675' = 'object_number=142, tag_tid=20, entry_tid=11558' 2024-08-31T22:26:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=143, tag_tid=20, entry_tid=11675' 2024-08-31T22:26:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=11675' 2024-08-31T22:26:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=11675' 2024-08-31T22:26:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11675 2024-08-31T22:26:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 11675 2024-08-31T22:26:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=11791], [object_number=142, tag_tid=20, entry_tid=11790], [object_number=141, tag_tid=20, entry_tid=11789], [object_number=140, tag_tid=20, entry_tid=11788]]], state=connected] 2024-08-31T22:26:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=11791], [object_number=142, tag_tid=20, entry_tid=11790], [object_number=141, tag_tid=20, entry_tid=11789], [object_number=140, tag_tid=20, entry_tid=11788]]], state=connected] 2024-08-31T22:26:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=143, tag_tid=20, entry_tid=11791' 2024-08-31T22:26:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=143, tag_tid=20, entry_tid=11791' 2024-08-31T22:26:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=11791' '!=' 'object_number=143, tag_tid=20, entry_tid=11675' 2024-08-31T22:26:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=11791' = 'object_number=143, tag_tid=20, entry_tid=11675' 2024-08-31T22:26:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=143, tag_tid=20, entry_tid=11791' 2024-08-31T22:26:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=11791' 2024-08-31T22:26:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=11791' 2024-08-31T22:26:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:40.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:40.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11791 2024-08-31T22:26:40.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 11791 2024-08-31T22:26:40.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:40.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:40.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:40.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:40.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:40.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:40.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:40.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=11890], [object_number=141, tag_tid=20, entry_tid=11889], [object_number=140, tag_tid=20, entry_tid=11888], [object_number=143, tag_tid=20, entry_tid=11887]]], state=connected] 2024-08-31T22:26:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=11890], [object_number=141, tag_tid=20, entry_tid=11889], [object_number=140, tag_tid=20, entry_tid=11888], [object_number=143, tag_tid=20, entry_tid=11887]]], state=connected] 2024-08-31T22:26:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=142, tag_tid=20, entry_tid=11890' 2024-08-31T22:26:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=142, tag_tid=20, entry_tid=11890' 2024-08-31T22:26:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=11890' '!=' 'object_number=143, tag_tid=20, entry_tid=11791' 2024-08-31T22:26:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=11890' = 'object_number=143, tag_tid=20, entry_tid=11791' 2024-08-31T22:26:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=142, tag_tid=20, entry_tid=11890' 2024-08-31T22:26:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=11890' 2024-08-31T22:26:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=11890' 2024-08-31T22:26:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11890 2024-08-31T22:26:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 11890 2024-08-31T22:26:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=11995], [object_number=142, tag_tid=20, entry_tid=11994], [object_number=141, tag_tid=20, entry_tid=11993], [object_number=140, tag_tid=20, entry_tid=11992]]], state=connected] 2024-08-31T22:26:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=11995], [object_number=142, tag_tid=20, entry_tid=11994], [object_number=141, tag_tid=20, entry_tid=11993], [object_number=140, tag_tid=20, entry_tid=11992]]], state=connected] 2024-08-31T22:26:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=143, tag_tid=20, entry_tid=11995' 2024-08-31T22:26:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=143, tag_tid=20, entry_tid=11995' 2024-08-31T22:26:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=11995' '!=' 'object_number=142, tag_tid=20, entry_tid=11890' 2024-08-31T22:26:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=11995' = 'object_number=142, tag_tid=20, entry_tid=11890' 2024-08-31T22:26:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=143, tag_tid=20, entry_tid=11995' 2024-08-31T22:26:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=11995' 2024-08-31T22:26:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=11995' 2024-08-31T22:26:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11995 2024-08-31T22:26:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 11995 2024-08-31T22:26:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=12100], [object_number=143, tag_tid=20, entry_tid=12099], [object_number=142, tag_tid=20, entry_tid=12098], [object_number=141, tag_tid=20, entry_tid=12097]]], state=connected] 2024-08-31T22:26:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=12100], [object_number=143, tag_tid=20, entry_tid=12099], [object_number=142, tag_tid=20, entry_tid=12098], [object_number=141, tag_tid=20, entry_tid=12097]]], state=connected] 2024-08-31T22:26:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=140, tag_tid=20, entry_tid=12100' 2024-08-31T22:26:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=140, tag_tid=20, entry_tid=12100' 2024-08-31T22:26:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=12100' '!=' 'object_number=143, tag_tid=20, entry_tid=11995' 2024-08-31T22:26:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=12100' = 'object_number=143, tag_tid=20, entry_tid=11995' 2024-08-31T22:26:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=140, tag_tid=20, entry_tid=12100' 2024-08-31T22:26:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=12100' 2024-08-31T22:26:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=12100' 2024-08-31T22:26:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12100 2024-08-31T22:26:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 12100 2024-08-31T22:26:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=141, tag_tid=20, entry_tid=12197], [object_number=140, tag_tid=20, entry_tid=12196], [object_number=143, tag_tid=20, entry_tid=12195], [object_number=142, tag_tid=20, entry_tid=12194]]], state=connected] 2024-08-31T22:26:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=141, tag_tid=20, entry_tid=12197], [object_number=140, tag_tid=20, entry_tid=12196], [object_number=143, tag_tid=20, entry_tid=12195], [object_number=142, tag_tid=20, entry_tid=12194]]], state=connected] 2024-08-31T22:26:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=141, tag_tid=20, entry_tid=12197' 2024-08-31T22:26:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=141, tag_tid=20, entry_tid=12197' 2024-08-31T22:26:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=141, tag_tid=20, entry_tid=12197' '!=' 'object_number=140, tag_tid=20, entry_tid=12100' 2024-08-31T22:26:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=141, tag_tid=20, entry_tid=12197' = 'object_number=140, tag_tid=20, entry_tid=12100' 2024-08-31T22:26:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=141, tag_tid=20, entry_tid=12197' 2024-08-31T22:26:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=141, tag_tid=20, entry_tid=12197' 2024-08-31T22:26:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=141, tag_tid=20, entry_tid=12197' 2024-08-31T22:26:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12197 2024-08-31T22:26:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 12197 2024-08-31T22:26:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:42.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:42.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:42.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:42.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:42.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:42.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:42.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:42.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:42.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:42.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:42.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:42.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:42.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:42.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:42.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:42.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:42.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:42.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:42.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:42.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=12327], [object_number=142, tag_tid=20, entry_tid=12326], [object_number=141, tag_tid=20, entry_tid=12325], [object_number=140, tag_tid=20, entry_tid=12324]]], state=connected] 2024-08-31T22:26:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=12327], [object_number=142, tag_tid=20, entry_tid=12326], [object_number=141, tag_tid=20, entry_tid=12325], [object_number=140, tag_tid=20, entry_tid=12324]]], state=connected] 2024-08-31T22:26:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=143, tag_tid=20, entry_tid=12327' 2024-08-31T22:26:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=143, tag_tid=20, entry_tid=12327' 2024-08-31T22:26:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=12327' '!=' 'object_number=141, tag_tid=20, entry_tid=12197' 2024-08-31T22:26:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=12327' = 'object_number=141, tag_tid=20, entry_tid=12197' 2024-08-31T22:26:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=143, tag_tid=20, entry_tid=12327' 2024-08-31T22:26:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=12327' 2024-08-31T22:26:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=12327' 2024-08-31T22:26:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12327 2024-08-31T22:26:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 12327 2024-08-31T22:26:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=12414], [object_number=141, tag_tid=20, entry_tid=12413], [object_number=140, tag_tid=20, entry_tid=12412], [object_number=143, tag_tid=20, entry_tid=12411]]], state=connected] 2024-08-31T22:26:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=12414], [object_number=141, tag_tid=20, entry_tid=12413], [object_number=140, tag_tid=20, entry_tid=12412], [object_number=143, tag_tid=20, entry_tid=12411]]], state=connected] 2024-08-31T22:26:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=142, tag_tid=20, entry_tid=12414' 2024-08-31T22:26:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=142, tag_tid=20, entry_tid=12414' 2024-08-31T22:26:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=12414' '!=' 'object_number=143, tag_tid=20, entry_tid=12327' 2024-08-31T22:26:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=12414' = 'object_number=143, tag_tid=20, entry_tid=12327' 2024-08-31T22:26:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=142, tag_tid=20, entry_tid=12414' 2024-08-31T22:26:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=12414' 2024-08-31T22:26:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:43.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=12414' 2024-08-31T22:26:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12414 2024-08-31T22:26:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 12414 2024-08-31T22:26:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:43.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=12503], [object_number=142, tag_tid=20, entry_tid=12502], [object_number=141, tag_tid=20, entry_tid=12501], [object_number=140, tag_tid=20, entry_tid=12500]]], state=connected] 2024-08-31T22:26:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=12503], [object_number=142, tag_tid=20, entry_tid=12502], [object_number=141, tag_tid=20, entry_tid=12501], [object_number=140, tag_tid=20, entry_tid=12500]]], state=connected] 2024-08-31T22:26:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=143, tag_tid=20, entry_tid=12503' 2024-08-31T22:26:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=143, tag_tid=20, entry_tid=12503' 2024-08-31T22:26:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=12503' '!=' 'object_number=142, tag_tid=20, entry_tid=12414' 2024-08-31T22:26:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=12503' = 'object_number=142, tag_tid=20, entry_tid=12414' 2024-08-31T22:26:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=143, tag_tid=20, entry_tid=12503' 2024-08-31T22:26:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:43.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=12503' 2024-08-31T22:26:43.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:43.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=12503' 2024-08-31T22:26:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12503 2024-08-31T22:26:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 12503 2024-08-31T22:26:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=12595], [object_number=142, tag_tid=20, entry_tid=12594], [object_number=141, tag_tid=20, entry_tid=12593], [object_number=140, tag_tid=20, entry_tid=12592]]], state=connected] 2024-08-31T22:26:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=12595], [object_number=142, tag_tid=20, entry_tid=12594], [object_number=141, tag_tid=20, entry_tid=12593], [object_number=140, tag_tid=20, entry_tid=12592]]], state=connected] 2024-08-31T22:26:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=143, tag_tid=20, entry_tid=12595' 2024-08-31T22:26:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=143, tag_tid=20, entry_tid=12595' 2024-08-31T22:26:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=12595' '!=' 'object_number=143, tag_tid=20, entry_tid=12503' 2024-08-31T22:26:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=12595' = 'object_number=143, tag_tid=20, entry_tid=12503' 2024-08-31T22:26:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=143, tag_tid=20, entry_tid=12595' 2024-08-31T22:26:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=12595' 2024-08-31T22:26:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=12595' 2024-08-31T22:26:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12595 2024-08-31T22:26:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 12595 2024-08-31T22:26:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=12706], [object_number=141, tag_tid=20, entry_tid=12705], [object_number=140, tag_tid=20, entry_tid=12704], [object_number=143, tag_tid=20, entry_tid=12703]]], state=connected] 2024-08-31T22:26:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=12706], [object_number=141, tag_tid=20, entry_tid=12705], [object_number=140, tag_tid=20, entry_tid=12704], [object_number=143, tag_tid=20, entry_tid=12703]]], state=connected] 2024-08-31T22:26:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=142, tag_tid=20, entry_tid=12706' 2024-08-31T22:26:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=142, tag_tid=20, entry_tid=12706' 2024-08-31T22:26:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=12706' '!=' 'object_number=143, tag_tid=20, entry_tid=12595' 2024-08-31T22:26:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=12706' = 'object_number=143, tag_tid=20, entry_tid=12595' 2024-08-31T22:26:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=142, tag_tid=20, entry_tid=12706' 2024-08-31T22:26:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=12706' 2024-08-31T22:26:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=12706' 2024-08-31T22:26:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12706 2024-08-31T22:26:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 12706 2024-08-31T22:26:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=12815], [object_number=142, tag_tid=20, entry_tid=12814], [object_number=141, tag_tid=20, entry_tid=12813], [object_number=140, tag_tid=20, entry_tid=12812]]], state=connected] 2024-08-31T22:26:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=12815], [object_number=142, tag_tid=20, entry_tid=12814], [object_number=141, tag_tid=20, entry_tid=12813], [object_number=140, tag_tid=20, entry_tid=12812]]], state=connected] 2024-08-31T22:26:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=143, tag_tid=20, entry_tid=12815' 2024-08-31T22:26:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=143, tag_tid=20, entry_tid=12815' 2024-08-31T22:26:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=12815' '!=' 'object_number=142, tag_tid=20, entry_tid=12706' 2024-08-31T22:26:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=12815' = 'object_number=142, tag_tid=20, entry_tid=12706' 2024-08-31T22:26:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=143, tag_tid=20, entry_tid=12815' 2024-08-31T22:26:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=12815' 2024-08-31T22:26:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:45.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:45.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:45.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=12815' 2024-08-31T22:26:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12815 2024-08-31T22:26:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 12815 2024-08-31T22:26:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:45.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=12910], [object_number=141, tag_tid=20, entry_tid=12909], [object_number=140, tag_tid=20, entry_tid=12908], [object_number=143, tag_tid=20, entry_tid=12907]]], state=connected] 2024-08-31T22:26:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=142, tag_tid=20, entry_tid=12910], [object_number=141, tag_tid=20, entry_tid=12909], [object_number=140, tag_tid=20, entry_tid=12908], [object_number=143, tag_tid=20, entry_tid=12907]]], state=connected] 2024-08-31T22:26:45.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=142, tag_tid=20, entry_tid=12910' 2024-08-31T22:26:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=142, tag_tid=20, entry_tid=12910' 2024-08-31T22:26:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=12910' '!=' 'object_number=143, tag_tid=20, entry_tid=12815' 2024-08-31T22:26:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=142, tag_tid=20, entry_tid=12910' = 'object_number=143, tag_tid=20, entry_tid=12815' 2024-08-31T22:26:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=142, tag_tid=20, entry_tid=12910' 2024-08-31T22:26:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=12910' 2024-08-31T22:26:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=142, tag_tid=20, entry_tid=12910' 2024-08-31T22:26:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12910 2024-08-31T22:26:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 12910 2024-08-31T22:26:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=12992], [object_number=143, tag_tid=20, entry_tid=12991], [object_number=142, tag_tid=20, entry_tid=12990], [object_number=141, tag_tid=20, entry_tid=12989]]], state=connected] 2024-08-31T22:26:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=12992], [object_number=143, tag_tid=20, entry_tid=12991], [object_number=142, tag_tid=20, entry_tid=12990], [object_number=141, tag_tid=20, entry_tid=12989]]], state=connected] 2024-08-31T22:26:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=140, tag_tid=20, entry_tid=12992' 2024-08-31T22:26:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=140, tag_tid=20, entry_tid=12992' 2024-08-31T22:26:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=12992' '!=' 'object_number=142, tag_tid=20, entry_tid=12910' 2024-08-31T22:26:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=12992' = 'object_number=142, tag_tid=20, entry_tid=12910' 2024-08-31T22:26:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=140, tag_tid=20, entry_tid=12992' 2024-08-31T22:26:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=12992' 2024-08-31T22:26:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=12992' 2024-08-31T22:26:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12992 2024-08-31T22:26:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 12992 2024-08-31T22:26:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:46.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=141, tag_tid=20, entry_tid=13069], [object_number=140, tag_tid=20, entry_tid=13068], [object_number=143, tag_tid=20, entry_tid=13067], [object_number=142, tag_tid=20, entry_tid=13066]]], state=connected] 2024-08-31T22:26:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=141, tag_tid=20, entry_tid=13069], [object_number=140, tag_tid=20, entry_tid=13068], [object_number=143, tag_tid=20, entry_tid=13067], [object_number=142, tag_tid=20, entry_tid=13066]]], state=connected] 2024-08-31T22:26:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=141, tag_tid=20, entry_tid=13069' 2024-08-31T22:26:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=141, tag_tid=20, entry_tid=13069' 2024-08-31T22:26:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=141, tag_tid=20, entry_tid=13069' '!=' 'object_number=140, tag_tid=20, entry_tid=12992' 2024-08-31T22:26:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=141, tag_tid=20, entry_tid=13069' = 'object_number=140, tag_tid=20, entry_tid=12992' 2024-08-31T22:26:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=141, tag_tid=20, entry_tid=13069' 2024-08-31T22:26:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=141, tag_tid=20, entry_tid=13069' 2024-08-31T22:26:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:46.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=141, tag_tid=20, entry_tid=13069' 2024-08-31T22:26:46.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:46.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13069 2024-08-31T22:26:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 13069 2024-08-31T22:26:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=141, tag_tid=20, entry_tid=13149], [object_number=140, tag_tid=20, entry_tid=13148], [object_number=143, tag_tid=20, entry_tid=13147], [object_number=142, tag_tid=20, entry_tid=13146]]], state=connected] 2024-08-31T22:26:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=141, tag_tid=20, entry_tid=13149], [object_number=140, tag_tid=20, entry_tid=13148], [object_number=143, tag_tid=20, entry_tid=13147], [object_number=142, tag_tid=20, entry_tid=13146]]], state=connected] 2024-08-31T22:26:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=141, tag_tid=20, entry_tid=13149' 2024-08-31T22:26:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=141, tag_tid=20, entry_tid=13149' 2024-08-31T22:26:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=141, tag_tid=20, entry_tid=13149' '!=' 'object_number=141, tag_tid=20, entry_tid=13069' 2024-08-31T22:26:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=141, tag_tid=20, entry_tid=13149' = 'object_number=141, tag_tid=20, entry_tid=13069' 2024-08-31T22:26:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=141, tag_tid=20, entry_tid=13149' 2024-08-31T22:26:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=141, tag_tid=20, entry_tid=13149' 2024-08-31T22:26:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=141, tag_tid=20, entry_tid=13149' 2024-08-31T22:26:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13149 2024-08-31T22:26:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 13149 2024-08-31T22:26:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=13228], [object_number=143, tag_tid=20, entry_tid=13227], [object_number=142, tag_tid=20, entry_tid=13226], [object_number=141, tag_tid=20, entry_tid=13225]]], state=connected] 2024-08-31T22:26:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=140, tag_tid=20, entry_tid=13228], [object_number=143, tag_tid=20, entry_tid=13227], [object_number=142, tag_tid=20, entry_tid=13226], [object_number=141, tag_tid=20, entry_tid=13225]]], state=connected] 2024-08-31T22:26:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=140, tag_tid=20, entry_tid=13228' 2024-08-31T22:26:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=140, tag_tid=20, entry_tid=13228' 2024-08-31T22:26:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=13228' '!=' 'object_number=141, tag_tid=20, entry_tid=13149' 2024-08-31T22:26:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=140, tag_tid=20, entry_tid=13228' = 'object_number=141, tag_tid=20, entry_tid=13149' 2024-08-31T22:26:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=140, tag_tid=20, entry_tid=13228' 2024-08-31T22:26:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=13228' 2024-08-31T22:26:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=140, tag_tid=20, entry_tid=13228' 2024-08-31T22:26:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13228 2024-08-31T22:26:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 13228 2024-08-31T22:26:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=13311], [object_number=142, tag_tid=20, entry_tid=13310], [object_number=141, tag_tid=20, entry_tid=13309], [object_number=140, tag_tid=20, entry_tid=13308]]], state=connected] 2024-08-31T22:26:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=13311], [object_number=142, tag_tid=20, entry_tid=13310], [object_number=141, tag_tid=20, entry_tid=13309], [object_number=140, tag_tid=20, entry_tid=13308]]], state=connected] 2024-08-31T22:26:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=143, tag_tid=20, entry_tid=13311' 2024-08-31T22:26:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=143, tag_tid=20, entry_tid=13311' 2024-08-31T22:26:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=13311' '!=' 'object_number=140, tag_tid=20, entry_tid=13228' 2024-08-31T22:26:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=13311' = 'object_number=140, tag_tid=20, entry_tid=13228' 2024-08-31T22:26:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=143, tag_tid=20, entry_tid=13311' 2024-08-31T22:26:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=13311' 2024-08-31T22:26:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=13311' 2024-08-31T22:26:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13311 2024-08-31T22:26:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 13311 2024-08-31T22:26:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:48.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:48.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:48.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=13363], [object_number=142, tag_tid=20, entry_tid=13362], [object_number=145, tag_tid=20, entry_tid=13361], [object_number=144, tag_tid=20, entry_tid=13360]]], state=connected] 2024-08-31T22:26:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=143, tag_tid=20, entry_tid=13363], [object_number=142, tag_tid=20, entry_tid=13362], [object_number=145, tag_tid=20, entry_tid=13361], [object_number=144, tag_tid=20, entry_tid=13360]]], state=connected] 2024-08-31T22:26:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=143, tag_tid=20, entry_tid=13363' 2024-08-31T22:26:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=143, tag_tid=20, entry_tid=13363' 2024-08-31T22:26:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=13363' '!=' 'object_number=143, tag_tid=20, entry_tid=13311' 2024-08-31T22:26:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=143, tag_tid=20, entry_tid=13363' = 'object_number=143, tag_tid=20, entry_tid=13311' 2024-08-31T22:26:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=143, tag_tid=20, entry_tid=13363' 2024-08-31T22:26:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=13363' 2024-08-31T22:26:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=143, tag_tid=20, entry_tid=13363' 2024-08-31T22:26:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13363 2024-08-31T22:26:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 13363 2024-08-31T22:26:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=145, tag_tid=20, entry_tid=13449], [object_number=144, tag_tid=20, entry_tid=13448], [object_number=147, tag_tid=20, entry_tid=13447], [object_number=142, tag_tid=20, entry_tid=13446]]], state=connected] 2024-08-31T22:26:49.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 35 2024-08-31T22:26:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=145, tag_tid=20, entry_tid=13449], [object_number=144, tag_tid=20, entry_tid=13448], [object_number=147, tag_tid=20, entry_tid=13447], [object_number=142, tag_tid=20, entry_tid=13446]]], state=connected] 2024-08-31T22:26:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=145, tag_tid=20, entry_tid=13449' 2024-08-31T22:26:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=145, tag_tid=20, entry_tid=13449' 2024-08-31T22:26:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=145, tag_tid=20, entry_tid=13449' '!=' 'object_number=143, tag_tid=20, entry_tid=13363' 2024-08-31T22:26:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=145, tag_tid=20, entry_tid=13449' = 'object_number=143, tag_tid=20, entry_tid=13363' 2024-08-31T22:26:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=145, tag_tid=20, entry_tid=13449' 2024-08-31T22:26:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=145, tag_tid=20, entry_tid=13449' 2024-08-31T22:26:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=145, tag_tid=20, entry_tid=13449' 2024-08-31T22:26:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13449 2024-08-31T22:26:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 13449 2024-08-31T22:26:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:49.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=145, tag_tid=20, entry_tid=13541], [object_number=144, tag_tid=20, entry_tid=13540], [object_number=147, tag_tid=20, entry_tid=13539], [object_number=146, tag_tid=20, entry_tid=13538]]], state=connected] 2024-08-31T22:26:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=145, tag_tid=20, entry_tid=13541], [object_number=144, tag_tid=20, entry_tid=13540], [object_number=147, tag_tid=20, entry_tid=13539], [object_number=146, tag_tid=20, entry_tid=13538]]], state=connected] 2024-08-31T22:26:50.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=145, tag_tid=20, entry_tid=13541' 2024-08-31T22:26:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=145, tag_tid=20, entry_tid=13541' 2024-08-31T22:26:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=145, tag_tid=20, entry_tid=13541' '!=' 'object_number=145, tag_tid=20, entry_tid=13449' 2024-08-31T22:26:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=145, tag_tid=20, entry_tid=13541' = 'object_number=145, tag_tid=20, entry_tid=13449' 2024-08-31T22:26:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=145, tag_tid=20, entry_tid=13541' 2024-08-31T22:26:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=145, tag_tid=20, entry_tid=13541' 2024-08-31T22:26:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=145, tag_tid=20, entry_tid=13541' 2024-08-31T22:26:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13541 2024-08-31T22:26:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 13541 2024-08-31T22:26:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:50.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:50.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:50.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:50.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:50.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=13632], [object_number=147, tag_tid=20, entry_tid=13631], [object_number=146, tag_tid=20, entry_tid=13630], [object_number=145, tag_tid=20, entry_tid=13629]]], state=connected] 2024-08-31T22:26:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=13632], [object_number=147, tag_tid=20, entry_tid=13631], [object_number=146, tag_tid=20, entry_tid=13630], [object_number=145, tag_tid=20, entry_tid=13629]]], state=connected] 2024-08-31T22:26:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=144, tag_tid=20, entry_tid=13632' 2024-08-31T22:26:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=144, tag_tid=20, entry_tid=13632' 2024-08-31T22:26:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=13632' '!=' 'object_number=145, tag_tid=20, entry_tid=13541' 2024-08-31T22:26:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=13632' = 'object_number=145, tag_tid=20, entry_tid=13541' 2024-08-31T22:26:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=144, tag_tid=20, entry_tid=13632' 2024-08-31T22:26:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=13632' 2024-08-31T22:26:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=13632' 2024-08-31T22:26:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13632 2024-08-31T22:26:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 13632 2024-08-31T22:26:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:50.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:50.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:50.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:51.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=147, tag_tid=20, entry_tid=13719], [object_number=146, tag_tid=20, entry_tid=13718], [object_number=145, tag_tid=20, entry_tid=13717], [object_number=144, tag_tid=20, entry_tid=13716]]], state=connected] 2024-08-31T22:26:51.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:51.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:51.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:51.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:51.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:51.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:51.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:51.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=147, tag_tid=20, entry_tid=13719], [object_number=146, tag_tid=20, entry_tid=13718], [object_number=145, tag_tid=20, entry_tid=13717], [object_number=144, tag_tid=20, entry_tid=13716]]], state=connected] 2024-08-31T22:26:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=147, tag_tid=20, entry_tid=13719' 2024-08-31T22:26:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=147, tag_tid=20, entry_tid=13719' 2024-08-31T22:26:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=147, tag_tid=20, entry_tid=13719' '!=' 'object_number=144, tag_tid=20, entry_tid=13632' 2024-08-31T22:26:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=147, tag_tid=20, entry_tid=13719' = 'object_number=144, tag_tid=20, entry_tid=13632' 2024-08-31T22:26:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=147, tag_tid=20, entry_tid=13719' 2024-08-31T22:26:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=147, tag_tid=20, entry_tid=13719' 2024-08-31T22:26:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:51.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=147, tag_tid=20, entry_tid=13719' 2024-08-31T22:26:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13719 2024-08-31T22:26:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 13719 2024-08-31T22:26:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:51.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:51.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:51.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:51.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:51.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=145, tag_tid=20, entry_tid=13813], [object_number=144, tag_tid=20, entry_tid=13812], [object_number=147, tag_tid=20, entry_tid=13811], [object_number=146, tag_tid=20, entry_tid=13810]]], state=connected] 2024-08-31T22:26:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=145, tag_tid=20, entry_tid=13813], [object_number=144, tag_tid=20, entry_tid=13812], [object_number=147, tag_tid=20, entry_tid=13811], [object_number=146, tag_tid=20, entry_tid=13810]]], state=connected] 2024-08-31T22:26:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=145, tag_tid=20, entry_tid=13813' 2024-08-31T22:26:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=145, tag_tid=20, entry_tid=13813' 2024-08-31T22:26:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=145, tag_tid=20, entry_tid=13813' '!=' 'object_number=147, tag_tid=20, entry_tid=13719' 2024-08-31T22:26:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=145, tag_tid=20, entry_tid=13813' = 'object_number=147, tag_tid=20, entry_tid=13719' 2024-08-31T22:26:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=145, tag_tid=20, entry_tid=13813' 2024-08-31T22:26:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=145, tag_tid=20, entry_tid=13813' 2024-08-31T22:26:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=145, tag_tid=20, entry_tid=13813' 2024-08-31T22:26:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13813 2024-08-31T22:26:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 13813 2024-08-31T22:26:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=147, tag_tid=20, entry_tid=13895], [object_number=146, tag_tid=20, entry_tid=13894], [object_number=145, tag_tid=20, entry_tid=13893], [object_number=144, tag_tid=20, entry_tid=13892]]], state=connected] 2024-08-31T22:26:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=147, tag_tid=20, entry_tid=13895], [object_number=146, tag_tid=20, entry_tid=13894], [object_number=145, tag_tid=20, entry_tid=13893], [object_number=144, tag_tid=20, entry_tid=13892]]], state=connected] 2024-08-31T22:26:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=147, tag_tid=20, entry_tid=13895' 2024-08-31T22:26:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=147, tag_tid=20, entry_tid=13895' 2024-08-31T22:26:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=147, tag_tid=20, entry_tid=13895' '!=' 'object_number=145, tag_tid=20, entry_tid=13813' 2024-08-31T22:26:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=147, tag_tid=20, entry_tid=13895' = 'object_number=145, tag_tid=20, entry_tid=13813' 2024-08-31T22:26:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=147, tag_tid=20, entry_tid=13895' 2024-08-31T22:26:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=147, tag_tid=20, entry_tid=13895' 2024-08-31T22:26:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=147, tag_tid=20, entry_tid=13895' 2024-08-31T22:26:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13895 2024-08-31T22:26:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 13895 2024-08-31T22:26:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:52.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:52.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=13980], [object_number=147, tag_tid=20, entry_tid=13979], [object_number=146, tag_tid=20, entry_tid=13978], [object_number=145, tag_tid=20, entry_tid=13977]]], state=connected] 2024-08-31T22:26:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:52.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:52.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:52.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:52.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:52.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:52.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:52.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:52.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:52.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:52.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:52.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=13980], [object_number=147, tag_tid=20, entry_tid=13979], [object_number=146, tag_tid=20, entry_tid=13978], [object_number=145, tag_tid=20, entry_tid=13977]]], state=connected] 2024-08-31T22:26:52.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=144, tag_tid=20, entry_tid=13980' 2024-08-31T22:26:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=144, tag_tid=20, entry_tid=13980' 2024-08-31T22:26:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=13980' '!=' 'object_number=147, tag_tid=20, entry_tid=13895' 2024-08-31T22:26:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=13980' = 'object_number=147, tag_tid=20, entry_tid=13895' 2024-08-31T22:26:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=144, tag_tid=20, entry_tid=13980' 2024-08-31T22:26:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=13980' 2024-08-31T22:26:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:52.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:52.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:52.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=13980' 2024-08-31T22:26:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13980 2024-08-31T22:26:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 13980 2024-08-31T22:26:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=14060], [object_number=147, tag_tid=20, entry_tid=14059], [object_number=146, tag_tid=20, entry_tid=14058], [object_number=145, tag_tid=20, entry_tid=14057]]], state=connected] 2024-08-31T22:26:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=14060], [object_number=147, tag_tid=20, entry_tid=14059], [object_number=146, tag_tid=20, entry_tid=14058], [object_number=145, tag_tid=20, entry_tid=14057]]], state=connected] 2024-08-31T22:26:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=144, tag_tid=20, entry_tid=14060' 2024-08-31T22:26:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=144, tag_tid=20, entry_tid=14060' 2024-08-31T22:26:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=14060' '!=' 'object_number=144, tag_tid=20, entry_tid=13980' 2024-08-31T22:26:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=14060' = 'object_number=144, tag_tid=20, entry_tid=13980' 2024-08-31T22:26:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=144, tag_tid=20, entry_tid=14060' 2024-08-31T22:26:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=14060' 2024-08-31T22:26:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=14060' 2024-08-31T22:26:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14060 2024-08-31T22:26:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 14060 2024-08-31T22:26:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=147, tag_tid=20, entry_tid=14147], [object_number=146, tag_tid=20, entry_tid=14146], [object_number=145, tag_tid=20, entry_tid=14145], [object_number=144, tag_tid=20, entry_tid=14144]]], state=connected] 2024-08-31T22:26:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=147, tag_tid=20, entry_tid=14147], [object_number=146, tag_tid=20, entry_tid=14146], [object_number=145, tag_tid=20, entry_tid=14145], [object_number=144, tag_tid=20, entry_tid=14144]]], state=connected] 2024-08-31T22:26:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=147, tag_tid=20, entry_tid=14147' 2024-08-31T22:26:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=147, tag_tid=20, entry_tid=14147' 2024-08-31T22:26:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=147, tag_tid=20, entry_tid=14147' '!=' 'object_number=144, tag_tid=20, entry_tid=14060' 2024-08-31T22:26:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=147, tag_tid=20, entry_tid=14147' = 'object_number=144, tag_tid=20, entry_tid=14060' 2024-08-31T22:26:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=147, tag_tid=20, entry_tid=14147' 2024-08-31T22:26:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=147, tag_tid=20, entry_tid=14147' 2024-08-31T22:26:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=147, tag_tid=20, entry_tid=14147' 2024-08-31T22:26:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14147 2024-08-31T22:26:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 14147 2024-08-31T22:26:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=147, tag_tid=20, entry_tid=14227], [object_number=146, tag_tid=20, entry_tid=14226], [object_number=145, tag_tid=20, entry_tid=14225], [object_number=144, tag_tid=20, entry_tid=14224]]], state=connected] 2024-08-31T22:26:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=147, tag_tid=20, entry_tid=14227], [object_number=146, tag_tid=20, entry_tid=14226], [object_number=145, tag_tid=20, entry_tid=14225], [object_number=144, tag_tid=20, entry_tid=14224]]], state=connected] 2024-08-31T22:26:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=147, tag_tid=20, entry_tid=14227' 2024-08-31T22:26:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=147, tag_tid=20, entry_tid=14227' 2024-08-31T22:26:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=147, tag_tid=20, entry_tid=14227' '!=' 'object_number=147, tag_tid=20, entry_tid=14147' 2024-08-31T22:26:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=147, tag_tid=20, entry_tid=14227' = 'object_number=147, tag_tid=20, entry_tid=14147' 2024-08-31T22:26:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=147, tag_tid=20, entry_tid=14227' 2024-08-31T22:26:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=147, tag_tid=20, entry_tid=14227' 2024-08-31T22:26:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=147, tag_tid=20, entry_tid=14227' 2024-08-31T22:26:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14227 2024-08-31T22:26:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 14227 2024-08-31T22:26:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:54.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:54.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=14316], [object_number=147, tag_tid=20, entry_tid=14315], [object_number=146, tag_tid=20, entry_tid=14314], [object_number=145, tag_tid=20, entry_tid=14313]]], state=connected] 2024-08-31T22:26:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=14316], [object_number=147, tag_tid=20, entry_tid=14315], [object_number=146, tag_tid=20, entry_tid=14314], [object_number=145, tag_tid=20, entry_tid=14313]]], state=connected] 2024-08-31T22:26:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=144, tag_tid=20, entry_tid=14316' 2024-08-31T22:26:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=144, tag_tid=20, entry_tid=14316' 2024-08-31T22:26:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=14316' '!=' 'object_number=147, tag_tid=20, entry_tid=14227' 2024-08-31T22:26:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=14316' = 'object_number=147, tag_tid=20, entry_tid=14227' 2024-08-31T22:26:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=144, tag_tid=20, entry_tid=14316' 2024-08-31T22:26:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=14316' 2024-08-31T22:26:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=14316' 2024-08-31T22:26:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14316 2024-08-31T22:26:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 14316 2024-08-31T22:26:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=146, tag_tid=20, entry_tid=14418], [object_number=145, tag_tid=20, entry_tid=14417], [object_number=144, tag_tid=20, entry_tid=14416], [object_number=147, tag_tid=20, entry_tid=14415]]], state=connected] 2024-08-31T22:26:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=146, tag_tid=20, entry_tid=14418], [object_number=145, tag_tid=20, entry_tid=14417], [object_number=144, tag_tid=20, entry_tid=14416], [object_number=147, tag_tid=20, entry_tid=14415]]], state=connected] 2024-08-31T22:26:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=146, tag_tid=20, entry_tid=14418' 2024-08-31T22:26:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=146, tag_tid=20, entry_tid=14418' 2024-08-31T22:26:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=146, tag_tid=20, entry_tid=14418' '!=' 'object_number=144, tag_tid=20, entry_tid=14316' 2024-08-31T22:26:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:55.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=146, tag_tid=20, entry_tid=14418' = 'object_number=144, tag_tid=20, entry_tid=14316' 2024-08-31T22:26:55.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=146, tag_tid=20, entry_tid=14418' 2024-08-31T22:26:55.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:55.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:55.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=146, tag_tid=20, entry_tid=14418' 2024-08-31T22:26:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=146, tag_tid=20, entry_tid=14418' 2024-08-31T22:26:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14418 2024-08-31T22:26:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 14418 2024-08-31T22:26:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:55.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:55.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=14520], [object_number=147, tag_tid=20, entry_tid=14519], [object_number=146, tag_tid=20, entry_tid=14518], [object_number=145, tag_tid=20, entry_tid=14517]]], state=connected] 2024-08-31T22:26:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=14520], [object_number=147, tag_tid=20, entry_tid=14519], [object_number=146, tag_tid=20, entry_tid=14518], [object_number=145, tag_tid=20, entry_tid=14517]]], state=connected] 2024-08-31T22:26:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=144, tag_tid=20, entry_tid=14520' 2024-08-31T22:26:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=144, tag_tid=20, entry_tid=14520' 2024-08-31T22:26:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=14520' '!=' 'object_number=146, tag_tid=20, entry_tid=14418' 2024-08-31T22:26:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=14520' = 'object_number=146, tag_tid=20, entry_tid=14418' 2024-08-31T22:26:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=144, tag_tid=20, entry_tid=14520' 2024-08-31T22:26:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=14520' 2024-08-31T22:26:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=14520' 2024-08-31T22:26:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14520 2024-08-31T22:26:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 14520 2024-08-31T22:26:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=146, tag_tid=20, entry_tid=14614], [object_number=145, tag_tid=20, entry_tid=14613], [object_number=144, tag_tid=20, entry_tid=14612], [object_number=147, tag_tid=20, entry_tid=14611]]], state=connected] 2024-08-31T22:26:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=146, tag_tid=20, entry_tid=14614], [object_number=145, tag_tid=20, entry_tid=14613], [object_number=144, tag_tid=20, entry_tid=14612], [object_number=147, tag_tid=20, entry_tid=14611]]], state=connected] 2024-08-31T22:26:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=146, tag_tid=20, entry_tid=14614' 2024-08-31T22:26:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=146, tag_tid=20, entry_tid=14614' 2024-08-31T22:26:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=146, tag_tid=20, entry_tid=14614' '!=' 'object_number=144, tag_tid=20, entry_tid=14520' 2024-08-31T22:26:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=146, tag_tid=20, entry_tid=14614' = 'object_number=144, tag_tid=20, entry_tid=14520' 2024-08-31T22:26:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=146, tag_tid=20, entry_tid=14614' 2024-08-31T22:26:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=146, tag_tid=20, entry_tid=14614' 2024-08-31T22:26:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=146, tag_tid=20, entry_tid=14614' 2024-08-31T22:26:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14614 2024-08-31T22:26:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 14614 2024-08-31T22:26:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:56.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:56.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:56.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:56.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:56.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:56.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:56.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:56.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:56.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=147, tag_tid=20, entry_tid=14699], [object_number=146, tag_tid=20, entry_tid=14698], [object_number=145, tag_tid=20, entry_tid=14697], [object_number=144, tag_tid=20, entry_tid=14696]]], state=connected] 2024-08-31T22:26:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=147, tag_tid=20, entry_tid=14699], [object_number=146, tag_tid=20, entry_tid=14698], [object_number=145, tag_tid=20, entry_tid=14697], [object_number=144, tag_tid=20, entry_tid=14696]]], state=connected] 2024-08-31T22:26:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=147, tag_tid=20, entry_tid=14699' 2024-08-31T22:26:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=147, tag_tid=20, entry_tid=14699' 2024-08-31T22:26:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=147, tag_tid=20, entry_tid=14699' '!=' 'object_number=146, tag_tid=20, entry_tid=14614' 2024-08-31T22:26:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=147, tag_tid=20, entry_tid=14699' = 'object_number=146, tag_tid=20, entry_tid=14614' 2024-08-31T22:26:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=147, tag_tid=20, entry_tid=14699' 2024-08-31T22:26:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=147, tag_tid=20, entry_tid=14699' 2024-08-31T22:26:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=147, tag_tid=20, entry_tid=14699' 2024-08-31T22:26:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14699 2024-08-31T22:26:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 14699 2024-08-31T22:26:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:57.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:57.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:57.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=145, tag_tid=20, entry_tid=14789], [object_number=144, tag_tid=20, entry_tid=14788], [object_number=147, tag_tid=20, entry_tid=14787], [object_number=146, tag_tid=20, entry_tid=14786]]], state=connected] 2024-08-31T22:26:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=145, tag_tid=20, entry_tid=14789], [object_number=144, tag_tid=20, entry_tid=14788], [object_number=147, tag_tid=20, entry_tid=14787], [object_number=146, tag_tid=20, entry_tid=14786]]], state=connected] 2024-08-31T22:26:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=145, tag_tid=20, entry_tid=14789' 2024-08-31T22:26:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=145, tag_tid=20, entry_tid=14789' 2024-08-31T22:26:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=145, tag_tid=20, entry_tid=14789' '!=' 'object_number=147, tag_tid=20, entry_tid=14699' 2024-08-31T22:26:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=145, tag_tid=20, entry_tid=14789' = 'object_number=147, tag_tid=20, entry_tid=14699' 2024-08-31T22:26:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=145, tag_tid=20, entry_tid=14789' 2024-08-31T22:26:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=145, tag_tid=20, entry_tid=14789' 2024-08-31T22:26:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=145, tag_tid=20, entry_tid=14789' 2024-08-31T22:26:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14789 2024-08-31T22:26:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 14789 2024-08-31T22:26:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=147, tag_tid=20, entry_tid=14879], [object_number=146, tag_tid=20, entry_tid=14878], [object_number=145, tag_tid=20, entry_tid=14877], [object_number=144, tag_tid=20, entry_tid=14876]]], state=connected] 2024-08-31T22:26:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=147, tag_tid=20, entry_tid=14879], [object_number=146, tag_tid=20, entry_tid=14878], [object_number=145, tag_tid=20, entry_tid=14877], [object_number=144, tag_tid=20, entry_tid=14876]]], state=connected] 2024-08-31T22:26:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=147, tag_tid=20, entry_tid=14879' 2024-08-31T22:26:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=147, tag_tid=20, entry_tid=14879' 2024-08-31T22:26:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=147, tag_tid=20, entry_tid=14879' '!=' 'object_number=145, tag_tid=20, entry_tid=14789' 2024-08-31T22:26:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=147, tag_tid=20, entry_tid=14879' = 'object_number=145, tag_tid=20, entry_tid=14789' 2024-08-31T22:26:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=147, tag_tid=20, entry_tid=14879' 2024-08-31T22:26:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=147, tag_tid=20, entry_tid=14879' 2024-08-31T22:26:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:57.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=147, tag_tid=20, entry_tid=14879' 2024-08-31T22:26:57.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:57.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14879 2024-08-31T22:26:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 14879 2024-08-31T22:26:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:58.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:58.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:58.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:58.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:58.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:58.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:58.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=147, tag_tid=20, entry_tid=14967], [object_number=146, tag_tid=20, entry_tid=14966], [object_number=145, tag_tid=20, entry_tid=14965], [object_number=144, tag_tid=20, entry_tid=14964]]], state=connected] 2024-08-31T22:26:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=147, tag_tid=20, entry_tid=14967], [object_number=146, tag_tid=20, entry_tid=14966], [object_number=145, tag_tid=20, entry_tid=14965], [object_number=144, tag_tid=20, entry_tid=14964]]], state=connected] 2024-08-31T22:26:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=147, tag_tid=20, entry_tid=14967' 2024-08-31T22:26:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=147, tag_tid=20, entry_tid=14967' 2024-08-31T22:26:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=147, tag_tid=20, entry_tid=14967' '!=' 'object_number=147, tag_tid=20, entry_tid=14879' 2024-08-31T22:26:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=147, tag_tid=20, entry_tid=14967' = 'object_number=147, tag_tid=20, entry_tid=14879' 2024-08-31T22:26:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=147, tag_tid=20, entry_tid=14967' 2024-08-31T22:26:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=147, tag_tid=20, entry_tid=14967' 2024-08-31T22:26:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=147, tag_tid=20, entry_tid=14967' 2024-08-31T22:26:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=14967 2024-08-31T22:26:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 14967 2024-08-31T22:26:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:58.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:58.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:58.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:58.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:58.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:58.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:58.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=145, tag_tid=20, entry_tid=15057], [object_number=144, tag_tid=20, entry_tid=15056], [object_number=147, tag_tid=20, entry_tid=15055], [object_number=146, tag_tid=20, entry_tid=15054]]], state=connected] 2024-08-31T22:26:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=145, tag_tid=20, entry_tid=15057], [object_number=144, tag_tid=20, entry_tid=15056], [object_number=147, tag_tid=20, entry_tid=15055], [object_number=146, tag_tid=20, entry_tid=15054]]], state=connected] 2024-08-31T22:26:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=145, tag_tid=20, entry_tid=15057' 2024-08-31T22:26:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=145, tag_tid=20, entry_tid=15057' 2024-08-31T22:26:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=145, tag_tid=20, entry_tid=15057' '!=' 'object_number=147, tag_tid=20, entry_tid=14967' 2024-08-31T22:26:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=145, tag_tid=20, entry_tid=15057' = 'object_number=147, tag_tid=20, entry_tid=14967' 2024-08-31T22:26:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=145, tag_tid=20, entry_tid=15057' 2024-08-31T22:26:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=145, tag_tid=20, entry_tid=15057' 2024-08-31T22:26:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=145, tag_tid=20, entry_tid=15057' 2024-08-31T22:26:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15057 2024-08-31T22:26:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 15057 2024-08-31T22:26:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:59.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:59.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=15148], [object_number=147, tag_tid=20, entry_tid=15147], [object_number=146, tag_tid=20, entry_tid=15146], [object_number=145, tag_tid=20, entry_tid=15145]]], state=connected] 2024-08-31T22:26:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:59.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:59.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=15148], [object_number=147, tag_tid=20, entry_tid=15147], [object_number=146, tag_tid=20, entry_tid=15146], [object_number=145, tag_tid=20, entry_tid=15145]]], state=connected] 2024-08-31T22:26:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=144, tag_tid=20, entry_tid=15148' 2024-08-31T22:26:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=144, tag_tid=20, entry_tid=15148' 2024-08-31T22:26:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=15148' '!=' 'object_number=145, tag_tid=20, entry_tid=15057' 2024-08-31T22:26:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=15148' = 'object_number=145, tag_tid=20, entry_tid=15057' 2024-08-31T22:26:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=144, tag_tid=20, entry_tid=15148' 2024-08-31T22:26:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=15148' 2024-08-31T22:26:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=15148' 2024-08-31T22:26:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15148 2024-08-31T22:26:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 15148 2024-08-31T22:26:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:59.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:26:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:26:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:26:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:26:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:26:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:26:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:26:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:26:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:26:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:26:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:26:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:26:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:26:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:26:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:26:59.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:26:59.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:26:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:26:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:26:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:26:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:26:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:26:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:26:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:26:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:26:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:26:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:26:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:26:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:26:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:26:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:26:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:26:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:26:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:26:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:26:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:26:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:26:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:26:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:26:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:26:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:26:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:26:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:26:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:26:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:26:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:26:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:26:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:26:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:26:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:26:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:59.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:59.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:26:59.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=146, tag_tid=20, entry_tid=15238], [object_number=145, tag_tid=20, entry_tid=15237], [object_number=144, tag_tid=20, entry_tid=15236], [object_number=147, tag_tid=20, entry_tid=15235]]], state=connected] 2024-08-31T22:26:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:26:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:26:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:26:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:26:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:26:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:26:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:26:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:26:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:26:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:26:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:59.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:26:59.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:26:59.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:26:59.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:26:59.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=146, tag_tid=20, entry_tid=15238], [object_number=145, tag_tid=20, entry_tid=15237], [object_number=144, tag_tid=20, entry_tid=15236], [object_number=147, tag_tid=20, entry_tid=15235]]], state=connected] 2024-08-31T22:26:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:26:59.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=146, tag_tid=20, entry_tid=15238' 2024-08-31T22:26:59.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=146, tag_tid=20, entry_tid=15238' 2024-08-31T22:26:59.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=146, tag_tid=20, entry_tid=15238' '!=' 'object_number=144, tag_tid=20, entry_tid=15148' 2024-08-31T22:26:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:26:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=146, tag_tid=20, entry_tid=15238' = 'object_number=144, tag_tid=20, entry_tid=15148' 2024-08-31T22:26:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=146, tag_tid=20, entry_tid=15238' 2024-08-31T22:26:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:26:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=146, tag_tid=20, entry_tid=15238' 2024-08-31T22:26:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:26:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:59.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:26:59.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:26:59.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:59.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:59.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:26:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=146, tag_tid=20, entry_tid=15238' 2024-08-31T22:26:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:26:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:26:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15238 2024-08-31T22:26:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 15238 2024-08-31T22:26:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:26:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:26:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:00.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=15332], [object_number=147, tag_tid=20, entry_tid=15331], [object_number=146, tag_tid=20, entry_tid=15330], [object_number=145, tag_tid=20, entry_tid=15329]]], state=connected] 2024-08-31T22:27:00.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=15332], [object_number=147, tag_tid=20, entry_tid=15331], [object_number=146, tag_tid=20, entry_tid=15330], [object_number=145, tag_tid=20, entry_tid=15329]]], state=connected] 2024-08-31T22:27:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=144, tag_tid=20, entry_tid=15332' 2024-08-31T22:27:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=144, tag_tid=20, entry_tid=15332' 2024-08-31T22:27:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=15332' '!=' 'object_number=146, tag_tid=20, entry_tid=15238' 2024-08-31T22:27:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=15332' = 'object_number=146, tag_tid=20, entry_tid=15238' 2024-08-31T22:27:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=144, tag_tid=20, entry_tid=15332' 2024-08-31T22:27:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=15332' 2024-08-31T22:27:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=15332' 2024-08-31T22:27:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15332 2024-08-31T22:27:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 15332 2024-08-31T22:27:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:00.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:00.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:00.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:00.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:00.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:00.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:00.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:00.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:00.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:00.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:00.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:00.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:00.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:00.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:00.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=15420], [object_number=147, tag_tid=20, entry_tid=15419], [object_number=146, tag_tid=20, entry_tid=15418], [object_number=145, tag_tid=20, entry_tid=15417]]], state=connected] 2024-08-31T22:27:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=15420], [object_number=147, tag_tid=20, entry_tid=15419], [object_number=146, tag_tid=20, entry_tid=15418], [object_number=145, tag_tid=20, entry_tid=15417]]], state=connected] 2024-08-31T22:27:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=144, tag_tid=20, entry_tid=15420' 2024-08-31T22:27:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=144, tag_tid=20, entry_tid=15420' 2024-08-31T22:27:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=15420' '!=' 'object_number=144, tag_tid=20, entry_tid=15332' 2024-08-31T22:27:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=15420' = 'object_number=144, tag_tid=20, entry_tid=15332' 2024-08-31T22:27:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=144, tag_tid=20, entry_tid=15420' 2024-08-31T22:27:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=15420' 2024-08-31T22:27:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=15420' 2024-08-31T22:27:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15420 2024-08-31T22:27:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 15420 2024-08-31T22:27:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:01.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:01.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:01.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:01.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:01.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:01.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:01.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:01.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=146, tag_tid=20, entry_tid=15514], [object_number=145, tag_tid=20, entry_tid=15513], [object_number=144, tag_tid=20, entry_tid=15512], [object_number=147, tag_tid=20, entry_tid=15511]]], state=connected] 2024-08-31T22:27:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:01.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:01.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:01.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:01.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:01.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:01.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=146, tag_tid=20, entry_tid=15514], [object_number=145, tag_tid=20, entry_tid=15513], [object_number=144, tag_tid=20, entry_tid=15512], [object_number=147, tag_tid=20, entry_tid=15511]]], state=connected] 2024-08-31T22:27:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=146, tag_tid=20, entry_tid=15514' 2024-08-31T22:27:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=146, tag_tid=20, entry_tid=15514' 2024-08-31T22:27:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=146, tag_tid=20, entry_tid=15514' '!=' 'object_number=144, tag_tid=20, entry_tid=15420' 2024-08-31T22:27:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=146, tag_tid=20, entry_tid=15514' = 'object_number=144, tag_tid=20, entry_tid=15420' 2024-08-31T22:27:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=146, tag_tid=20, entry_tid=15514' 2024-08-31T22:27:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=146, tag_tid=20, entry_tid=15514' 2024-08-31T22:27:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:01.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:01.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:01.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:01.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=146, tag_tid=20, entry_tid=15514' 2024-08-31T22:27:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15514 2024-08-31T22:27:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 15514 2024-08-31T22:27:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:01.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:01.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:01.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:01.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=15604], [object_number=147, tag_tid=20, entry_tid=15603], [object_number=146, tag_tid=20, entry_tid=15602], [object_number=145, tag_tid=20, entry_tid=15601]]], state=connected] 2024-08-31T22:27:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:01.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:01.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:01.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:01.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:01.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:01.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=15604], [object_number=147, tag_tid=20, entry_tid=15603], [object_number=146, tag_tid=20, entry_tid=15602], [object_number=145, tag_tid=20, entry_tid=15601]]], state=connected] 2024-08-31T22:27:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=144, tag_tid=20, entry_tid=15604' 2024-08-31T22:27:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=144, tag_tid=20, entry_tid=15604' 2024-08-31T22:27:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=15604' '!=' 'object_number=146, tag_tid=20, entry_tid=15514' 2024-08-31T22:27:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=15604' = 'object_number=146, tag_tid=20, entry_tid=15514' 2024-08-31T22:27:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=144, tag_tid=20, entry_tid=15604' 2024-08-31T22:27:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=15604' 2024-08-31T22:27:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=15604' 2024-08-31T22:27:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15604 2024-08-31T22:27:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 15604 2024-08-31T22:27:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:02.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:02.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:02.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=15692], [object_number=147, tag_tid=20, entry_tid=15691], [object_number=146, tag_tid=20, entry_tid=15690], [object_number=145, tag_tid=20, entry_tid=15689]]], state=connected] 2024-08-31T22:27:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=15692], [object_number=147, tag_tid=20, entry_tid=15691], [object_number=146, tag_tid=20, entry_tid=15690], [object_number=145, tag_tid=20, entry_tid=15689]]], state=connected] 2024-08-31T22:27:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=144, tag_tid=20, entry_tid=15692' 2024-08-31T22:27:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=144, tag_tid=20, entry_tid=15692' 2024-08-31T22:27:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=15692' '!=' 'object_number=144, tag_tid=20, entry_tid=15604' 2024-08-31T22:27:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=15692' = 'object_number=144, tag_tid=20, entry_tid=15604' 2024-08-31T22:27:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=144, tag_tid=20, entry_tid=15692' 2024-08-31T22:27:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=15692' 2024-08-31T22:27:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=15692' 2024-08-31T22:27:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15692 2024-08-31T22:27:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 15692 2024-08-31T22:27:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:02.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:02.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:02.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:02.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=146, tag_tid=20, entry_tid=15774], [object_number=145, tag_tid=20, entry_tid=15773], [object_number=144, tag_tid=20, entry_tid=15772], [object_number=147, tag_tid=20, entry_tid=15771]]], state=connected] 2024-08-31T22:27:02.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:02.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:02.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:02.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:02.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:02.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:02.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:02.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=146, tag_tid=20, entry_tid=15774], [object_number=145, tag_tid=20, entry_tid=15773], [object_number=144, tag_tid=20, entry_tid=15772], [object_number=147, tag_tid=20, entry_tid=15771]]], state=connected] 2024-08-31T22:27:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=146, tag_tid=20, entry_tid=15774' 2024-08-31T22:27:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=146, tag_tid=20, entry_tid=15774' 2024-08-31T22:27:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=146, tag_tid=20, entry_tid=15774' '!=' 'object_number=144, tag_tid=20, entry_tid=15692' 2024-08-31T22:27:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=146, tag_tid=20, entry_tid=15774' = 'object_number=144, tag_tid=20, entry_tid=15692' 2024-08-31T22:27:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=146, tag_tid=20, entry_tid=15774' 2024-08-31T22:27:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=146, tag_tid=20, entry_tid=15774' 2024-08-31T22:27:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=146, tag_tid=20, entry_tid=15774' 2024-08-31T22:27:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15774 2024-08-31T22:27:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 15774 2024-08-31T22:27:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=145, tag_tid=20, entry_tid=15857], [object_number=144, tag_tid=20, entry_tid=15856], [object_number=147, tag_tid=20, entry_tid=15855], [object_number=146, tag_tid=20, entry_tid=15854]]], state=connected] 2024-08-31T22:27:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=145, tag_tid=20, entry_tid=15857], [object_number=144, tag_tid=20, entry_tid=15856], [object_number=147, tag_tid=20, entry_tid=15855], [object_number=146, tag_tid=20, entry_tid=15854]]], state=connected] 2024-08-31T22:27:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=145, tag_tid=20, entry_tid=15857' 2024-08-31T22:27:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=145, tag_tid=20, entry_tid=15857' 2024-08-31T22:27:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=145, tag_tid=20, entry_tid=15857' '!=' 'object_number=146, tag_tid=20, entry_tid=15774' 2024-08-31T22:27:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=145, tag_tid=20, entry_tid=15857' = 'object_number=146, tag_tid=20, entry_tid=15774' 2024-08-31T22:27:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=145, tag_tid=20, entry_tid=15857' 2024-08-31T22:27:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=145, tag_tid=20, entry_tid=15857' 2024-08-31T22:27:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=145, tag_tid=20, entry_tid=15857' 2024-08-31T22:27:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15857 2024-08-31T22:27:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 15857 2024-08-31T22:27:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:03.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:03.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=15940], [object_number=147, tag_tid=20, entry_tid=15939], [object_number=146, tag_tid=20, entry_tid=15938], [object_number=145, tag_tid=20, entry_tid=15937]]], state=connected] 2024-08-31T22:27:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:03.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:03.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:03.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:03.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:03.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=15940], [object_number=147, tag_tid=20, entry_tid=15939], [object_number=146, tag_tid=20, entry_tid=15938], [object_number=145, tag_tid=20, entry_tid=15937]]], state=connected] 2024-08-31T22:27:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=144, tag_tid=20, entry_tid=15940' 2024-08-31T22:27:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=144, tag_tid=20, entry_tid=15940' 2024-08-31T22:27:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=15940' '!=' 'object_number=145, tag_tid=20, entry_tid=15857' 2024-08-31T22:27:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=15940' = 'object_number=145, tag_tid=20, entry_tid=15857' 2024-08-31T22:27:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=144, tag_tid=20, entry_tid=15940' 2024-08-31T22:27:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=15940' 2024-08-31T22:27:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=15940' 2024-08-31T22:27:03.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:03.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:03.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=15940 2024-08-31T22:27:03.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 15940 2024-08-31T22:27:03.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:03.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:03.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:04.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=147, tag_tid=20, entry_tid=16027], [object_number=146, tag_tid=20, entry_tid=16026], [object_number=145, tag_tid=20, entry_tid=16025], [object_number=144, tag_tid=20, entry_tid=16024]]], state=connected] 2024-08-31T22:27:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=147, tag_tid=20, entry_tid=16027], [object_number=146, tag_tid=20, entry_tid=16026], [object_number=145, tag_tid=20, entry_tid=16025], [object_number=144, tag_tid=20, entry_tid=16024]]], state=connected] 2024-08-31T22:27:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=147, tag_tid=20, entry_tid=16027' 2024-08-31T22:27:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=147, tag_tid=20, entry_tid=16027' 2024-08-31T22:27:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=147, tag_tid=20, entry_tid=16027' '!=' 'object_number=144, tag_tid=20, entry_tid=15940' 2024-08-31T22:27:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=147, tag_tid=20, entry_tid=16027' = 'object_number=144, tag_tid=20, entry_tid=15940' 2024-08-31T22:27:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=147, tag_tid=20, entry_tid=16027' 2024-08-31T22:27:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=147, tag_tid=20, entry_tid=16027' 2024-08-31T22:27:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:04.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=147, tag_tid=20, entry_tid=16027' 2024-08-31T22:27:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16027 2024-08-31T22:27:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 16027 2024-08-31T22:27:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:04.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:04.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=145, tag_tid=20, entry_tid=16113], [object_number=144, tag_tid=20, entry_tid=16112], [object_number=147, tag_tid=20, entry_tid=16111], [object_number=146, tag_tid=20, entry_tid=16110]]], state=connected] 2024-08-31T22:27:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=145, tag_tid=20, entry_tid=16113], [object_number=144, tag_tid=20, entry_tid=16112], [object_number=147, tag_tid=20, entry_tid=16111], [object_number=146, tag_tid=20, entry_tid=16110]]], state=connected] 2024-08-31T22:27:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=145, tag_tid=20, entry_tid=16113' 2024-08-31T22:27:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=145, tag_tid=20, entry_tid=16113' 2024-08-31T22:27:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=145, tag_tid=20, entry_tid=16113' '!=' 'object_number=147, tag_tid=20, entry_tid=16027' 2024-08-31T22:27:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=145, tag_tid=20, entry_tid=16113' = 'object_number=147, tag_tid=20, entry_tid=16027' 2024-08-31T22:27:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=145, tag_tid=20, entry_tid=16113' 2024-08-31T22:27:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=145, tag_tid=20, entry_tid=16113' 2024-08-31T22:27:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=145, tag_tid=20, entry_tid=16113' 2024-08-31T22:27:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16113 2024-08-31T22:27:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 16113 2024-08-31T22:27:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:05.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:05.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=145, tag_tid=20, entry_tid=16201], [object_number=144, tag_tid=20, entry_tid=16200], [object_number=147, tag_tid=20, entry_tid=16199], [object_number=146, tag_tid=20, entry_tid=16198]]], state=connected] 2024-08-31T22:27:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=145, tag_tid=20, entry_tid=16201], [object_number=144, tag_tid=20, entry_tid=16200], [object_number=147, tag_tid=20, entry_tid=16199], [object_number=146, tag_tid=20, entry_tid=16198]]], state=connected] 2024-08-31T22:27:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=145, tag_tid=20, entry_tid=16201' 2024-08-31T22:27:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=145, tag_tid=20, entry_tid=16201' 2024-08-31T22:27:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=145, tag_tid=20, entry_tid=16201' '!=' 'object_number=145, tag_tid=20, entry_tid=16113' 2024-08-31T22:27:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=145, tag_tid=20, entry_tid=16201' = 'object_number=145, tag_tid=20, entry_tid=16113' 2024-08-31T22:27:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=145, tag_tid=20, entry_tid=16201' 2024-08-31T22:27:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=145, tag_tid=20, entry_tid=16201' 2024-08-31T22:27:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=145, tag_tid=20, entry_tid=16201' 2024-08-31T22:27:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:05.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16201 2024-08-31T22:27:05.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 16201 2024-08-31T22:27:05.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:05.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:05.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=145, tag_tid=20, entry_tid=16285], [object_number=144, tag_tid=20, entry_tid=16284], [object_number=147, tag_tid=20, entry_tid=16283], [object_number=146, tag_tid=20, entry_tid=16282]]], state=connected] 2024-08-31T22:27:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=145, tag_tid=20, entry_tid=16285], [object_number=144, tag_tid=20, entry_tid=16284], [object_number=147, tag_tid=20, entry_tid=16283], [object_number=146, tag_tid=20, entry_tid=16282]]], state=connected] 2024-08-31T22:27:05.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=145, tag_tid=20, entry_tid=16285' 2024-08-31T22:27:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=145, tag_tid=20, entry_tid=16285' 2024-08-31T22:27:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=145, tag_tid=20, entry_tid=16285' '!=' 'object_number=145, tag_tid=20, entry_tid=16201' 2024-08-31T22:27:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=145, tag_tid=20, entry_tid=16285' = 'object_number=145, tag_tid=20, entry_tid=16201' 2024-08-31T22:27:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=145, tag_tid=20, entry_tid=16285' 2024-08-31T22:27:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=145, tag_tid=20, entry_tid=16285' 2024-08-31T22:27:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=145, tag_tid=20, entry_tid=16285' 2024-08-31T22:27:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16285 2024-08-31T22:27:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 16285 2024-08-31T22:27:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=16372], [object_number=147, tag_tid=20, entry_tid=16371], [object_number=146, tag_tid=20, entry_tid=16370], [object_number=145, tag_tid=20, entry_tid=16369]]], state=connected] 2024-08-31T22:27:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=16372], [object_number=147, tag_tid=20, entry_tid=16371], [object_number=146, tag_tid=20, entry_tid=16370], [object_number=145, tag_tid=20, entry_tid=16369]]], state=connected] 2024-08-31T22:27:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=144, tag_tid=20, entry_tid=16372' 2024-08-31T22:27:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=144, tag_tid=20, entry_tid=16372' 2024-08-31T22:27:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=16372' '!=' 'object_number=145, tag_tid=20, entry_tid=16285' 2024-08-31T22:27:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=16372' = 'object_number=145, tag_tid=20, entry_tid=16285' 2024-08-31T22:27:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=144, tag_tid=20, entry_tid=16372' 2024-08-31T22:27:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=16372' 2024-08-31T22:27:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=16372' 2024-08-31T22:27:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16372 2024-08-31T22:27:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 16372 2024-08-31T22:27:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:06.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:06.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:06.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:06.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:06.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:06.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:06.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:06.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:06.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:06.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:06.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:06.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=16456], [object_number=147, tag_tid=20, entry_tid=16455], [object_number=146, tag_tid=20, entry_tid=16454], [object_number=145, tag_tid=20, entry_tid=16453]]], state=connected] 2024-08-31T22:27:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=144, tag_tid=20, entry_tid=16456], [object_number=147, tag_tid=20, entry_tid=16455], [object_number=146, tag_tid=20, entry_tid=16454], [object_number=145, tag_tid=20, entry_tid=16453]]], state=connected] 2024-08-31T22:27:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=144, tag_tid=20, entry_tid=16456' 2024-08-31T22:27:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=144, tag_tid=20, entry_tid=16456' 2024-08-31T22:27:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=16456' '!=' 'object_number=144, tag_tid=20, entry_tid=16372' 2024-08-31T22:27:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=144, tag_tid=20, entry_tid=16456' = 'object_number=144, tag_tid=20, entry_tid=16372' 2024-08-31T22:27:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=144, tag_tid=20, entry_tid=16456' 2024-08-31T22:27:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=16456' 2024-08-31T22:27:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=144, tag_tid=20, entry_tid=16456' 2024-08-31T22:27:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16456 2024-08-31T22:27:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 16456 2024-08-31T22:27:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=146, tag_tid=20, entry_tid=16546], [object_number=145, tag_tid=20, entry_tid=16545], [object_number=144, tag_tid=20, entry_tid=16544], [object_number=147, tag_tid=20, entry_tid=16543]]], state=connected] 2024-08-31T22:27:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:07.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:07.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:07.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:07.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:07.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=146, tag_tid=20, entry_tid=16546], [object_number=145, tag_tid=20, entry_tid=16545], [object_number=144, tag_tid=20, entry_tid=16544], [object_number=147, tag_tid=20, entry_tid=16543]]], state=connected] 2024-08-31T22:27:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=146, tag_tid=20, entry_tid=16546' 2024-08-31T22:27:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=146, tag_tid=20, entry_tid=16546' 2024-08-31T22:27:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=146, tag_tid=20, entry_tid=16546' '!=' 'object_number=144, tag_tid=20, entry_tid=16456' 2024-08-31T22:27:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=146, tag_tid=20, entry_tid=16546' = 'object_number=144, tag_tid=20, entry_tid=16456' 2024-08-31T22:27:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=146, tag_tid=20, entry_tid=16546' 2024-08-31T22:27:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=146, tag_tid=20, entry_tid=16546' 2024-08-31T22:27:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=146, tag_tid=20, entry_tid=16546' 2024-08-31T22:27:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16546 2024-08-31T22:27:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 16546 2024-08-31T22:27:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=146, tag_tid=20, entry_tid=16630], [object_number=145, tag_tid=20, entry_tid=16629], [object_number=144, tag_tid=20, entry_tid=16628], [object_number=147, tag_tid=20, entry_tid=16627]]], state=connected] 2024-08-31T22:27:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:07.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:07.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=146, tag_tid=20, entry_tid=16630], [object_number=145, tag_tid=20, entry_tid=16629], [object_number=144, tag_tid=20, entry_tid=16628], [object_number=147, tag_tid=20, entry_tid=16627]]], state=connected] 2024-08-31T22:27:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=146, tag_tid=20, entry_tid=16630' 2024-08-31T22:27:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=146, tag_tid=20, entry_tid=16630' 2024-08-31T22:27:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=146, tag_tid=20, entry_tid=16630' '!=' 'object_number=146, tag_tid=20, entry_tid=16546' 2024-08-31T22:27:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=146, tag_tid=20, entry_tid=16630' = 'object_number=146, tag_tid=20, entry_tid=16546' 2024-08-31T22:27:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=146, tag_tid=20, entry_tid=16630' 2024-08-31T22:27:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=146, tag_tid=20, entry_tid=16630' 2024-08-31T22:27:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=146, tag_tid=20, entry_tid=16630' 2024-08-31T22:27:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16630 2024-08-31T22:27:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 16630 2024-08-31T22:27:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:08.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=147, tag_tid=20, entry_tid=16719], [object_number=146, tag_tid=20, entry_tid=16718], [object_number=145, tag_tid=20, entry_tid=16717], [object_number=144, tag_tid=20, entry_tid=16716]]], state=connected] 2024-08-31T22:27:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=147, tag_tid=20, entry_tid=16719], [object_number=146, tag_tid=20, entry_tid=16718], [object_number=145, tag_tid=20, entry_tid=16717], [object_number=144, tag_tid=20, entry_tid=16716]]], state=connected] 2024-08-31T22:27:08.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=147, tag_tid=20, entry_tid=16719' 2024-08-31T22:27:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=147, tag_tid=20, entry_tid=16719' 2024-08-31T22:27:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=147, tag_tid=20, entry_tid=16719' '!=' 'object_number=146, tag_tid=20, entry_tid=16630' 2024-08-31T22:27:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=147, tag_tid=20, entry_tid=16719' = 'object_number=146, tag_tid=20, entry_tid=16630' 2024-08-31T22:27:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=147, tag_tid=20, entry_tid=16719' 2024-08-31T22:27:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=147, tag_tid=20, entry_tid=16719' 2024-08-31T22:27:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=147, tag_tid=20, entry_tid=16719' 2024-08-31T22:27:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16719 2024-08-31T22:27:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 16719 2024-08-31T22:27:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=146, tag_tid=20, entry_tid=16818], [object_number=149, tag_tid=20, entry_tid=16817], [object_number=148, tag_tid=20, entry_tid=16816], [object_number=151, tag_tid=20, entry_tid=16815]]], state=connected] 2024-08-31T22:27:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 36 2024-08-31T22:27:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=146, tag_tid=20, entry_tid=16818], [object_number=149, tag_tid=20, entry_tid=16817], [object_number=148, tag_tid=20, entry_tid=16816], [object_number=151, tag_tid=20, entry_tid=16815]]], state=connected] 2024-08-31T22:27:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=146, tag_tid=20, entry_tid=16818' 2024-08-31T22:27:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=146, tag_tid=20, entry_tid=16818' 2024-08-31T22:27:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=146, tag_tid=20, entry_tid=16818' '!=' 'object_number=147, tag_tid=20, entry_tid=16719' 2024-08-31T22:27:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=146, tag_tid=20, entry_tid=16818' = 'object_number=147, tag_tid=20, entry_tid=16719' 2024-08-31T22:27:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=146, tag_tid=20, entry_tid=16818' 2024-08-31T22:27:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=146, tag_tid=20, entry_tid=16818' 2024-08-31T22:27:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:09.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:09.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=146, tag_tid=20, entry_tid=16818' 2024-08-31T22:27:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16818 2024-08-31T22:27:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 16818 2024-08-31T22:27:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=148, tag_tid=20, entry_tid=16912], [object_number=151, tag_tid=20, entry_tid=16911], [object_number=150, tag_tid=20, entry_tid=16910], [object_number=149, tag_tid=20, entry_tid=16909]]], state=connected] 2024-08-31T22:27:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=148, tag_tid=20, entry_tid=16912], [object_number=151, tag_tid=20, entry_tid=16911], [object_number=150, tag_tid=20, entry_tid=16910], [object_number=149, tag_tid=20, entry_tid=16909]]], state=connected] 2024-08-31T22:27:09.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:09.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=148, tag_tid=20, entry_tid=16912' 2024-08-31T22:27:09.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=148, tag_tid=20, entry_tid=16912' 2024-08-31T22:27:09.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=148, tag_tid=20, entry_tid=16912' '!=' 'object_number=146, tag_tid=20, entry_tid=16818' 2024-08-31T22:27:09.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:09.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=148, tag_tid=20, entry_tid=16912' = 'object_number=146, tag_tid=20, entry_tid=16818' 2024-08-31T22:27:09.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=148, tag_tid=20, entry_tid=16912' 2024-08-31T22:27:09.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:09.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:09.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:09.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=148, tag_tid=20, entry_tid=16912' 2024-08-31T22:27:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=148, tag_tid=20, entry_tid=16912' 2024-08-31T22:27:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16912 2024-08-31T22:27:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 16912 2024-08-31T22:27:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:09.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:09.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=149, tag_tid=20, entry_tid=16969], [object_number=148, tag_tid=20, entry_tid=16968], [object_number=151, tag_tid=20, entry_tid=16967], [object_number=150, tag_tid=20, entry_tid=16966]]], state=connected] 2024-08-31T22:27:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=149, tag_tid=20, entry_tid=16969], [object_number=148, tag_tid=20, entry_tid=16968], [object_number=151, tag_tid=20, entry_tid=16967], [object_number=150, tag_tid=20, entry_tid=16966]]], state=connected] 2024-08-31T22:27:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=149, tag_tid=20, entry_tid=16969' 2024-08-31T22:27:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=149, tag_tid=20, entry_tid=16969' 2024-08-31T22:27:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=149, tag_tid=20, entry_tid=16969' '!=' 'object_number=148, tag_tid=20, entry_tid=16912' 2024-08-31T22:27:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=149, tag_tid=20, entry_tid=16969' = 'object_number=148, tag_tid=20, entry_tid=16912' 2024-08-31T22:27:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=149, tag_tid=20, entry_tid=16969' 2024-08-31T22:27:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=20, entry_tid=16969' 2024-08-31T22:27:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=20, entry_tid=16969' 2024-08-31T22:27:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=16969 2024-08-31T22:27:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 16969 2024-08-31T22:27:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:10.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=150, tag_tid=20, entry_tid=17066], [object_number=149, tag_tid=20, entry_tid=17065], [object_number=148, tag_tid=20, entry_tid=17064], [object_number=151, tag_tid=20, entry_tid=17063]]], state=connected] 2024-08-31T22:27:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=150, tag_tid=20, entry_tid=17066], [object_number=149, tag_tid=20, entry_tid=17065], [object_number=148, tag_tid=20, entry_tid=17064], [object_number=151, tag_tid=20, entry_tid=17063]]], state=connected] 2024-08-31T22:27:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=150, tag_tid=20, entry_tid=17066' 2024-08-31T22:27:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=150, tag_tid=20, entry_tid=17066' 2024-08-31T22:27:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=150, tag_tid=20, entry_tid=17066' '!=' 'object_number=149, tag_tid=20, entry_tid=16969' 2024-08-31T22:27:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=150, tag_tid=20, entry_tid=17066' = 'object_number=149, tag_tid=20, entry_tid=16969' 2024-08-31T22:27:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=150, tag_tid=20, entry_tid=17066' 2024-08-31T22:27:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=150, tag_tid=20, entry_tid=17066' 2024-08-31T22:27:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=150, tag_tid=20, entry_tid=17066' 2024-08-31T22:27:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17066 2024-08-31T22:27:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 17066 2024-08-31T22:27:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:10.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:10.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=150, tag_tid=20, entry_tid=17150], [object_number=149, tag_tid=20, entry_tid=17149], [object_number=148, tag_tid=20, entry_tid=17148], [object_number=151, tag_tid=20, entry_tid=17147]]], state=connected] 2024-08-31T22:27:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=150, tag_tid=20, entry_tid=17150], [object_number=149, tag_tid=20, entry_tid=17149], [object_number=148, tag_tid=20, entry_tid=17148], [object_number=151, tag_tid=20, entry_tid=17147]]], state=connected] 2024-08-31T22:27:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=150, tag_tid=20, entry_tid=17150' 2024-08-31T22:27:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=150, tag_tid=20, entry_tid=17150' 2024-08-31T22:27:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=150, tag_tid=20, entry_tid=17150' '!=' 'object_number=150, tag_tid=20, entry_tid=17066' 2024-08-31T22:27:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=150, tag_tid=20, entry_tid=17150' = 'object_number=150, tag_tid=20, entry_tid=17066' 2024-08-31T22:27:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=150, tag_tid=20, entry_tid=17150' 2024-08-31T22:27:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:10.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=150, tag_tid=20, entry_tid=17150' 2024-08-31T22:27:10.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:10.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=150, tag_tid=20, entry_tid=17150' 2024-08-31T22:27:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17150 2024-08-31T22:27:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 17150 2024-08-31T22:27:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:11.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:11.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=149, tag_tid=20, entry_tid=17237], [object_number=148, tag_tid=20, entry_tid=17236], [object_number=151, tag_tid=20, entry_tid=17235], [object_number=150, tag_tid=20, entry_tid=17234]]], state=connected] 2024-08-31T22:27:11.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:11.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:11.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:11.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:11.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:11.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:11.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=149, tag_tid=20, entry_tid=17237], [object_number=148, tag_tid=20, entry_tid=17236], [object_number=151, tag_tid=20, entry_tid=17235], [object_number=150, tag_tid=20, entry_tid=17234]]], state=connected] 2024-08-31T22:27:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=149, tag_tid=20, entry_tid=17237' 2024-08-31T22:27:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=149, tag_tid=20, entry_tid=17237' 2024-08-31T22:27:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=149, tag_tid=20, entry_tid=17237' '!=' 'object_number=150, tag_tid=20, entry_tid=17150' 2024-08-31T22:27:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=149, tag_tid=20, entry_tid=17237' = 'object_number=150, tag_tid=20, entry_tid=17150' 2024-08-31T22:27:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=149, tag_tid=20, entry_tid=17237' 2024-08-31T22:27:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=20, entry_tid=17237' 2024-08-31T22:27:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:11.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=20, entry_tid=17237' 2024-08-31T22:27:11.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:11.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17237 2024-08-31T22:27:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 17237 2024-08-31T22:27:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=148, tag_tid=20, entry_tid=17332], [object_number=151, tag_tid=20, entry_tid=17331], [object_number=150, tag_tid=20, entry_tid=17330], [object_number=149, tag_tid=20, entry_tid=17329]]], state=connected] 2024-08-31T22:27:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:11.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=148, tag_tid=20, entry_tid=17332], [object_number=151, tag_tid=20, entry_tid=17331], [object_number=150, tag_tid=20, entry_tid=17330], [object_number=149, tag_tid=20, entry_tid=17329]]], state=connected] 2024-08-31T22:27:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=148, tag_tid=20, entry_tid=17332' 2024-08-31T22:27:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=148, tag_tid=20, entry_tid=17332' 2024-08-31T22:27:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=148, tag_tid=20, entry_tid=17332' '!=' 'object_number=149, tag_tid=20, entry_tid=17237' 2024-08-31T22:27:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=148, tag_tid=20, entry_tid=17332' = 'object_number=149, tag_tid=20, entry_tid=17237' 2024-08-31T22:27:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=148, tag_tid=20, entry_tid=17332' 2024-08-31T22:27:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:11.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:11.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=148, tag_tid=20, entry_tid=17332' 2024-08-31T22:27:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:11.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:11.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:11.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:11.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:11.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=148, tag_tid=20, entry_tid=17332' 2024-08-31T22:27:11.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:11.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17332 2024-08-31T22:27:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 17332 2024-08-31T22:27:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:12.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:12.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:12.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:12.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:12.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:12.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:12.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=148, tag_tid=20, entry_tid=17424], [object_number=151, tag_tid=20, entry_tid=17423], [object_number=150, tag_tid=20, entry_tid=17422], [object_number=149, tag_tid=20, entry_tid=17421]]], state=connected] 2024-08-31T22:27:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=148, tag_tid=20, entry_tid=17424], [object_number=151, tag_tid=20, entry_tid=17423], [object_number=150, tag_tid=20, entry_tid=17422], [object_number=149, tag_tid=20, entry_tid=17421]]], state=connected] 2024-08-31T22:27:12.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=148, tag_tid=20, entry_tid=17424' 2024-08-31T22:27:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=148, tag_tid=20, entry_tid=17424' 2024-08-31T22:27:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=148, tag_tid=20, entry_tid=17424' '!=' 'object_number=148, tag_tid=20, entry_tid=17332' 2024-08-31T22:27:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=148, tag_tid=20, entry_tid=17424' = 'object_number=148, tag_tid=20, entry_tid=17332' 2024-08-31T22:27:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=148, tag_tid=20, entry_tid=17424' 2024-08-31T22:27:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=148, tag_tid=20, entry_tid=17424' 2024-08-31T22:27:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:12.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:12.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:12.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=148, tag_tid=20, entry_tid=17424' 2024-08-31T22:27:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17424 2024-08-31T22:27:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 17424 2024-08-31T22:27:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=149, tag_tid=20, entry_tid=17517], [object_number=148, tag_tid=20, entry_tid=17516], [object_number=151, tag_tid=20, entry_tid=17515], [object_number=150, tag_tid=20, entry_tid=17514]]], state=connected] 2024-08-31T22:27:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=149, tag_tid=20, entry_tid=17517], [object_number=148, tag_tid=20, entry_tid=17516], [object_number=151, tag_tid=20, entry_tid=17515], [object_number=150, tag_tid=20, entry_tid=17514]]], state=connected] 2024-08-31T22:27:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=149, tag_tid=20, entry_tid=17517' 2024-08-31T22:27:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=149, tag_tid=20, entry_tid=17517' 2024-08-31T22:27:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=149, tag_tid=20, entry_tid=17517' '!=' 'object_number=148, tag_tid=20, entry_tid=17424' 2024-08-31T22:27:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=149, tag_tid=20, entry_tid=17517' = 'object_number=148, tag_tid=20, entry_tid=17424' 2024-08-31T22:27:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=149, tag_tid=20, entry_tid=17517' 2024-08-31T22:27:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=20, entry_tid=17517' 2024-08-31T22:27:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=20, entry_tid=17517' 2024-08-31T22:27:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17517 2024-08-31T22:27:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 17517 2024-08-31T22:27:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=148, tag_tid=20, entry_tid=17608], [object_number=151, tag_tid=20, entry_tid=17607], [object_number=150, tag_tid=20, entry_tid=17606], [object_number=149, tag_tid=20, entry_tid=17605]]], state=connected] 2024-08-31T22:27:13.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:13.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:13.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:13.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=148, tag_tid=20, entry_tid=17608], [object_number=151, tag_tid=20, entry_tid=17607], [object_number=150, tag_tid=20, entry_tid=17606], [object_number=149, tag_tid=20, entry_tid=17605]]], state=connected] 2024-08-31T22:27:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=148, tag_tid=20, entry_tid=17608' 2024-08-31T22:27:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=148, tag_tid=20, entry_tid=17608' 2024-08-31T22:27:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=148, tag_tid=20, entry_tid=17608' '!=' 'object_number=149, tag_tid=20, entry_tid=17517' 2024-08-31T22:27:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=148, tag_tid=20, entry_tid=17608' = 'object_number=149, tag_tid=20, entry_tid=17517' 2024-08-31T22:27:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=148, tag_tid=20, entry_tid=17608' 2024-08-31T22:27:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:13.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:13.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=148, tag_tid=20, entry_tid=17608' 2024-08-31T22:27:13.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:13.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:13.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:13.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:13.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:13.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=148, tag_tid=20, entry_tid=17608' 2024-08-31T22:27:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17608 2024-08-31T22:27:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 17608 2024-08-31T22:27:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=149, tag_tid=20, entry_tid=17697], [object_number=148, tag_tid=20, entry_tid=17696], [object_number=151, tag_tid=20, entry_tid=17695], [object_number=150, tag_tid=20, entry_tid=17694]]], state=connected] 2024-08-31T22:27:13.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:13.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:13.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:13.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:13.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=149, tag_tid=20, entry_tid=17697], [object_number=148, tag_tid=20, entry_tid=17696], [object_number=151, tag_tid=20, entry_tid=17695], [object_number=150, tag_tid=20, entry_tid=17694]]], state=connected] 2024-08-31T22:27:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=149, tag_tid=20, entry_tid=17697' 2024-08-31T22:27:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=149, tag_tid=20, entry_tid=17697' 2024-08-31T22:27:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=149, tag_tid=20, entry_tid=17697' '!=' 'object_number=148, tag_tid=20, entry_tid=17608' 2024-08-31T22:27:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=149, tag_tid=20, entry_tid=17697' = 'object_number=148, tag_tid=20, entry_tid=17608' 2024-08-31T22:27:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=149, tag_tid=20, entry_tid=17697' 2024-08-31T22:27:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=20, entry_tid=17697' 2024-08-31T22:27:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=20, entry_tid=17697' 2024-08-31T22:27:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17697 2024-08-31T22:27:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 17697 2024-08-31T22:27:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=149, tag_tid=20, entry_tid=17789], [object_number=148, tag_tid=20, entry_tid=17788], [object_number=151, tag_tid=20, entry_tid=17787], [object_number=150, tag_tid=20, entry_tid=17786]]], state=connected] 2024-08-31T22:27:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=149, tag_tid=20, entry_tid=17789], [object_number=148, tag_tid=20, entry_tid=17788], [object_number=151, tag_tid=20, entry_tid=17787], [object_number=150, tag_tid=20, entry_tid=17786]]], state=connected] 2024-08-31T22:27:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=149, tag_tid=20, entry_tid=17789' 2024-08-31T22:27:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=149, tag_tid=20, entry_tid=17789' 2024-08-31T22:27:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=149, tag_tid=20, entry_tid=17789' '!=' 'object_number=149, tag_tid=20, entry_tid=17697' 2024-08-31T22:27:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=149, tag_tid=20, entry_tid=17789' = 'object_number=149, tag_tid=20, entry_tid=17697' 2024-08-31T22:27:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=149, tag_tid=20, entry_tid=17789' 2024-08-31T22:27:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=20, entry_tid=17789' 2024-08-31T22:27:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=20, entry_tid=17789' 2024-08-31T22:27:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17789 2024-08-31T22:27:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 17789 2024-08-31T22:27:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:14.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:14.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:14.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:14.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:14.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=149, tag_tid=20, entry_tid=17881], [object_number=148, tag_tid=20, entry_tid=17880], [object_number=151, tag_tid=20, entry_tid=17879], [object_number=150, tag_tid=20, entry_tid=17878]]], state=connected] 2024-08-31T22:27:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:14.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:14.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:14.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:14.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:14.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:14.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=149, tag_tid=20, entry_tid=17881], [object_number=148, tag_tid=20, entry_tid=17880], [object_number=151, tag_tid=20, entry_tid=17879], [object_number=150, tag_tid=20, entry_tid=17878]]], state=connected] 2024-08-31T22:27:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=149, tag_tid=20, entry_tid=17881' 2024-08-31T22:27:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=149, tag_tid=20, entry_tid=17881' 2024-08-31T22:27:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=149, tag_tid=20, entry_tid=17881' '!=' 'object_number=149, tag_tid=20, entry_tid=17789' 2024-08-31T22:27:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=149, tag_tid=20, entry_tid=17881' = 'object_number=149, tag_tid=20, entry_tid=17789' 2024-08-31T22:27:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=149, tag_tid=20, entry_tid=17881' 2024-08-31T22:27:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=20, entry_tid=17881' 2024-08-31T22:27:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=20, entry_tid=17881' 2024-08-31T22:27:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17881 2024-08-31T22:27:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 17881 2024-08-31T22:27:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:15.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:15.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:15.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=151, tag_tid=20, entry_tid=17975], [object_number=150, tag_tid=20, entry_tid=17974], [object_number=149, tag_tid=20, entry_tid=17973], [object_number=148, tag_tid=20, entry_tid=17972]]], state=connected] 2024-08-31T22:27:15.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:15.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=151, tag_tid=20, entry_tid=17975], [object_number=150, tag_tid=20, entry_tid=17974], [object_number=149, tag_tid=20, entry_tid=17973], [object_number=148, tag_tid=20, entry_tid=17972]]], state=connected] 2024-08-31T22:27:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=151, tag_tid=20, entry_tid=17975' 2024-08-31T22:27:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=151, tag_tid=20, entry_tid=17975' 2024-08-31T22:27:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=151, tag_tid=20, entry_tid=17975' '!=' 'object_number=149, tag_tid=20, entry_tid=17881' 2024-08-31T22:27:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=151, tag_tid=20, entry_tid=17975' = 'object_number=149, tag_tid=20, entry_tid=17881' 2024-08-31T22:27:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=151, tag_tid=20, entry_tid=17975' 2024-08-31T22:27:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=151, tag_tid=20, entry_tid=17975' 2024-08-31T22:27:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=151, tag_tid=20, entry_tid=17975' 2024-08-31T22:27:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=17975 2024-08-31T22:27:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 17975 2024-08-31T22:27:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:15.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:15.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:15.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:15.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:15.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:15.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=151, tag_tid=20, entry_tid=18067], [object_number=150, tag_tid=20, entry_tid=18066], [object_number=149, tag_tid=20, entry_tid=18065], [object_number=148, tag_tid=20, entry_tid=18064]]], state=connected] 2024-08-31T22:27:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=151, tag_tid=20, entry_tid=18067], [object_number=150, tag_tid=20, entry_tid=18066], [object_number=149, tag_tid=20, entry_tid=18065], [object_number=148, tag_tid=20, entry_tid=18064]]], state=connected] 2024-08-31T22:27:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=151, tag_tid=20, entry_tid=18067' 2024-08-31T22:27:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=151, tag_tid=20, entry_tid=18067' 2024-08-31T22:27:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=151, tag_tid=20, entry_tid=18067' '!=' 'object_number=151, tag_tid=20, entry_tid=17975' 2024-08-31T22:27:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=151, tag_tid=20, entry_tid=18067' = 'object_number=151, tag_tid=20, entry_tid=17975' 2024-08-31T22:27:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=151, tag_tid=20, entry_tid=18067' 2024-08-31T22:27:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=151, tag_tid=20, entry_tid=18067' 2024-08-31T22:27:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=151, tag_tid=20, entry_tid=18067' 2024-08-31T22:27:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18067 2024-08-31T22:27:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 18067 2024-08-31T22:27:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=151, tag_tid=20, entry_tid=18151], [object_number=150, tag_tid=20, entry_tid=18150], [object_number=149, tag_tid=20, entry_tid=18149], [object_number=148, tag_tid=20, entry_tid=18148]]], state=connected] 2024-08-31T22:27:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=151, tag_tid=20, entry_tid=18151], [object_number=150, tag_tid=20, entry_tid=18150], [object_number=149, tag_tid=20, entry_tid=18149], [object_number=148, tag_tid=20, entry_tid=18148]]], state=connected] 2024-08-31T22:27:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=151, tag_tid=20, entry_tid=18151' 2024-08-31T22:27:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=151, tag_tid=20, entry_tid=18151' 2024-08-31T22:27:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=151, tag_tid=20, entry_tid=18151' '!=' 'object_number=151, tag_tid=20, entry_tid=18067' 2024-08-31T22:27:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=151, tag_tid=20, entry_tid=18151' = 'object_number=151, tag_tid=20, entry_tid=18067' 2024-08-31T22:27:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=151, tag_tid=20, entry_tid=18151' 2024-08-31T22:27:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=151, tag_tid=20, entry_tid=18151' 2024-08-31T22:27:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=151, tag_tid=20, entry_tid=18151' 2024-08-31T22:27:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:16.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18151 2024-08-31T22:27:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 18151 2024-08-31T22:27:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=150, tag_tid=20, entry_tid=18238], [object_number=149, tag_tid=20, entry_tid=18237], [object_number=148, tag_tid=20, entry_tid=18236], [object_number=151, tag_tid=20, entry_tid=18235]]], state=connected] 2024-08-31T22:27:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=150, tag_tid=20, entry_tid=18238], [object_number=149, tag_tid=20, entry_tid=18237], [object_number=148, tag_tid=20, entry_tid=18236], [object_number=151, tag_tid=20, entry_tid=18235]]], state=connected] 2024-08-31T22:27:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=150, tag_tid=20, entry_tid=18238' 2024-08-31T22:27:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=150, tag_tid=20, entry_tid=18238' 2024-08-31T22:27:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=150, tag_tid=20, entry_tid=18238' '!=' 'object_number=151, tag_tid=20, entry_tid=18151' 2024-08-31T22:27:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=150, tag_tid=20, entry_tid=18238' = 'object_number=151, tag_tid=20, entry_tid=18151' 2024-08-31T22:27:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=150, tag_tid=20, entry_tid=18238' 2024-08-31T22:27:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=150, tag_tid=20, entry_tid=18238' 2024-08-31T22:27:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:17.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:17.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:17.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:17.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=150, tag_tid=20, entry_tid=18238' 2024-08-31T22:27:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18238 2024-08-31T22:27:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 18238 2024-08-31T22:27:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=148, tag_tid=20, entry_tid=18324], [object_number=151, tag_tid=20, entry_tid=18323], [object_number=150, tag_tid=20, entry_tid=18322], [object_number=149, tag_tid=20, entry_tid=18321]]], state=connected] 2024-08-31T22:27:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=148, tag_tid=20, entry_tid=18324], [object_number=151, tag_tid=20, entry_tid=18323], [object_number=150, tag_tid=20, entry_tid=18322], [object_number=149, tag_tid=20, entry_tid=18321]]], state=connected] 2024-08-31T22:27:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=148, tag_tid=20, entry_tid=18324' 2024-08-31T22:27:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=148, tag_tid=20, entry_tid=18324' 2024-08-31T22:27:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=148, tag_tid=20, entry_tid=18324' '!=' 'object_number=150, tag_tid=20, entry_tid=18238' 2024-08-31T22:27:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=148, tag_tid=20, entry_tid=18324' = 'object_number=150, tag_tid=20, entry_tid=18238' 2024-08-31T22:27:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=148, tag_tid=20, entry_tid=18324' 2024-08-31T22:27:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=148, tag_tid=20, entry_tid=18324' 2024-08-31T22:27:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=148, tag_tid=20, entry_tid=18324' 2024-08-31T22:27:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18324 2024-08-31T22:27:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 18324 2024-08-31T22:27:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:17.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:17.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:17.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:17.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:17.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:17.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:17.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:17.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:17.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=150, tag_tid=20, entry_tid=18414], [object_number=149, tag_tid=20, entry_tid=18413], [object_number=148, tag_tid=20, entry_tid=18412], [object_number=151, tag_tid=20, entry_tid=18411]]], state=connected] 2024-08-31T22:27:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=150, tag_tid=20, entry_tid=18414], [object_number=149, tag_tid=20, entry_tid=18413], [object_number=148, tag_tid=20, entry_tid=18412], [object_number=151, tag_tid=20, entry_tid=18411]]], state=connected] 2024-08-31T22:27:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=150, tag_tid=20, entry_tid=18414' 2024-08-31T22:27:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=150, tag_tid=20, entry_tid=18414' 2024-08-31T22:27:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=150, tag_tid=20, entry_tid=18414' '!=' 'object_number=148, tag_tid=20, entry_tid=18324' 2024-08-31T22:27:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=150, tag_tid=20, entry_tid=18414' = 'object_number=148, tag_tid=20, entry_tid=18324' 2024-08-31T22:27:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=150, tag_tid=20, entry_tid=18414' 2024-08-31T22:27:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=150, tag_tid=20, entry_tid=18414' 2024-08-31T22:27:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=150, tag_tid=20, entry_tid=18414' 2024-08-31T22:27:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18414 2024-08-31T22:27:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 18414 2024-08-31T22:27:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:18.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:18.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:18.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:18.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:18.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:18.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:18.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:18.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:18.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=150, tag_tid=20, entry_tid=18502], [object_number=149, tag_tid=20, entry_tid=18501], [object_number=148, tag_tid=20, entry_tid=18500], [object_number=151, tag_tid=20, entry_tid=18499]]], state=connected] 2024-08-31T22:27:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=150, tag_tid=20, entry_tid=18502], [object_number=149, tag_tid=20, entry_tid=18501], [object_number=148, tag_tid=20, entry_tid=18500], [object_number=151, tag_tid=20, entry_tid=18499]]], state=connected] 2024-08-31T22:27:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=150, tag_tid=20, entry_tid=18502' 2024-08-31T22:27:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=150, tag_tid=20, entry_tid=18502' 2024-08-31T22:27:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=150, tag_tid=20, entry_tid=18502' '!=' 'object_number=150, tag_tid=20, entry_tid=18414' 2024-08-31T22:27:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=150, tag_tid=20, entry_tid=18502' = 'object_number=150, tag_tid=20, entry_tid=18414' 2024-08-31T22:27:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=150, tag_tid=20, entry_tid=18502' 2024-08-31T22:27:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:18.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=150, tag_tid=20, entry_tid=18502' 2024-08-31T22:27:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:18.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=150, tag_tid=20, entry_tid=18502' 2024-08-31T22:27:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18502 2024-08-31T22:27:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 18502 2024-08-31T22:27:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=149, tag_tid=20, entry_tid=18573], [object_number=148, tag_tid=20, entry_tid=18572], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:19.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:19.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:19.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:19.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:19.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=149, tag_tid=20, entry_tid=18573], [object_number=148, tag_tid=20, entry_tid=18572], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:19.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=149, tag_tid=20, entry_tid=18573' 2024-08-31T22:27:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=149, tag_tid=20, entry_tid=18573' 2024-08-31T22:27:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=149, tag_tid=20, entry_tid=18573' '!=' 'object_number=150, tag_tid=20, entry_tid=18502' 2024-08-31T22:27:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:27:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=149, tag_tid=20, entry_tid=18573' = 'object_number=150, tag_tid=20, entry_tid=18502' 2024-08-31T22:27:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=149, tag_tid=20, entry_tid=18573' 2024-08-31T22:27:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=21 2024-08-31T22:27:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=20, entry_tid=18573' 2024-08-31T22:27:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:27:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=20 2024-08-31T22:27:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:27:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=149, tag_tid=20, entry_tid=18573' 2024-08-31T22:27:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:27:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:27:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=18573 2024-08-31T22:27:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 21 = 20 -a 1 -le 18573 2024-08-31T22:27:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:27:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:27:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=149, tag_tid=20, entry_tid=18573], [object_number=148, tag_tid=20, entry_tid=18572], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=149, tag_tid=20, entry_tid=18573], [object_number=148, tag_tid=20, entry_tid=18572], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=149, tag_tid=20, entry_tid=18573' 2024-08-31T22:27:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=149, tag_tid=20, entry_tid=18573' 2024-08-31T22:27:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=149, tag_tid=20, entry_tid=18573' '!=' 'object_number=149, tag_tid=20, entry_tid=18573' 2024-08-31T22:27:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:27:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.4 2024-08-31T22:27:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:27:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:27:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:27:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:27:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:27:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:27:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:27:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:27:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:27:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:27:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:27:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:27:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:27:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:27:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:27:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:27:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:27:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:27:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:27:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:27:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:27:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:27:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:27:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:27:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:27:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:27:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:27:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:27:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:27:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:27:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:27:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:27:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:27:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:27:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:27:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:27:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:27:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:27:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:27:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:27:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:27:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:27:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:27:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:27:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:27:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:27:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:27:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:27:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:27:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:27:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:27:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:27:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:27:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:27:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:27:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 37 2024-08-31T22:27:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 37 2024-08-31T22:27:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:27:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=149, tag_tid=21, entry_tid=1], [object_number=148, tag_tid=21, entry_tid=0], [object_number=151, tag_tid=20, entry_tid=18571], [object_number=150, tag_tid=20, entry_tid=18570]]], state=connected] 2024-08-31T22:27:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:27:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=149, tag_tid=21, entry_tid=1' -a 'object_number=149, tag_tid=21, entry_tid=1' = 'object_number=149, tag_tid=21, entry_tid=1' 2024-08-31T22:27:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-08-31T22:27:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap7 2024-08-31T22:27:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:27:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:27:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:27:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-08-31T22:27:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'test': 2024-08-31T22:27:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:27:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:27:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 13 2024-08-31T22:27:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a87b4c6a1b 2024-08-31T22:27:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a87b4c6a1b 2024-08-31T22:27:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:27:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:27:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:27:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:27:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:14:57 2024 2024-08-31T22:27:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:24:32 2024 2024-08-31T22:27:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:27:09 2024 2024-08-31T22:27:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:27:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a87b4c6a1b 2024-08-31T22:27:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:27:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:27:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:27:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:27:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n snap6-clean ']' 2024-08-31T22:27:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap6-clean 2024-08-31T22:27:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap6-clean 2024-08-31T22:27:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:27:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:27:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:27:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:27:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:27:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:27:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:27:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:27:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:27:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:27:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap7 2024-08-31T22:27:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap7 2024-08-31T22:27:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:27:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:27:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:27:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:27:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:27:24.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:27:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:27:26.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:27:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:27:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:27:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ clean_snap_name=snap7-clean 2024-08-31T22:27:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-08-31T22:27:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:27:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap7-clean 2024-08-31T22:27:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-08-31T22:27:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:27:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror test 2024-08-31T22:27:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:27:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:27:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=11 2024-08-31T22:27:26.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:27:26.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-08-31T22:27:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 74810368 bytes 2024-08-31T22:27:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 139573248 bytes 2024-08-31T22:27:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 164434944 bytes 2024-08-31T22:27:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 184151040 bytes 2024-08-31T22:27:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:500 IOs, 200722432 bytes 2024-08-31T22:27:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:600 IOs, 206315520 bytes 2024-08-31T22:27:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:27:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:27:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:27:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-08-31T22:27:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:27:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:27:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:27:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:27:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:27:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-08-31T22:27:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:27:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:27:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:27:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:27:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:27:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-08-31T22:27:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='test: 2024-08-31T22:27:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: global_id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:27:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+replaying 2024-08-31T22:27:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: replaying, {"bytes_per_second":2699942.37,"entries_behind_primary":0,"entries_per_second":168.2,"non_primary_position":{"entry_tid":1,"object_number":149,"tag_tid":21},"primary_position":{"entry_tid":1,"object_number":149,"tag_tid":21}} 2024-08-31T22:27:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: service: mirror.0 on smithi032 2024-08-31T22:27:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:27:25 2024-08-31T22:27:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: peer_sites: 2024-08-31T22:27:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: name: cluster2 2024-08-31T22:27:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+stopped 2024-08-31T22:27:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: local image is primary 2024-08-31T22:27:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:27:35' 2024-08-31T22:27:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ state: .*up+replaying' 2024-08-31T22:27:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: state: up+replaying 2024-08-31T22:27:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ description: .*' 2024-08-31T22:27:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: description: replaying, {"bytes_per_second":2699942.37,"entries_behind_primary":0,"entries_per_second":168.2,"non_primary_position":{"entry_tid":1,"object_number":149,"tag_tid":21},"primary_position":{"entry_tid":1,"object_number":149,"tag_tid":21}} 2024-08-31T22:27:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:27:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:27:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^up+' 2024-08-31T22:27:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:27:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep 'service: *mirror..* on ' 2024-08-31T22:27:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: service: mirror.0 on smithi032 2024-08-31T22:27:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update 2024-08-31T22:27:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-08-31T22:27:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:27:25' 2024-08-31T22:27:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-08-31 22:27:25' 2024-08-31T22:27:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:27:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:27:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'prev_last_update=2024-08-31 22:27:25' 2024-08-31T22:27:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:27:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:27:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-08-31T22:27:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='OKOKOK142574264mirror.0smithi03219.1.1-244-g3091dfc0trueOK42614267mirror.1smithi03219.1.1-244-g3091dfc0falseOKtestdd2c705e-25cf-4565-981f-d48de48eee84up+replayingreplaying, {"bytes_per_second":2699942.37,"entries_behind_primary":0,"entries_per_second":168.2,"non_primary_position":{"entry_tid":1,"object_number":149,"tag_tid":21},"primary_position":{"entry_tid":1,"object_number":149,"tag_tid":21}}42574264mirror.0smithi0322024-08-31 22:27:25cluster24ca32b00-2164-486e-837e-e93795a16af1up+stoppedlocal image is primary2024-08-31 22:27:35' 2024-08-31T22:27:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update state 2024-08-31T22:27:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-08-31T22:27:39.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:27:25' 2024-08-31T22:27:39.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-08-31T22:27:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ state=up+replaying 2024-08-31T22:27:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep up+replaying 2024-08-31T22:27:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:27:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:27:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:27:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap8 2024-08-31T22:27:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap8 2024-08-31T22:27:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:27:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:27:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:27:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap8 2024-08-31T22:27:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:28:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-08-31T22:28:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-08-31T22:28:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:28:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:28:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:28:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror test started 2024-08-31T22:28:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:28:12.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:28:12.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:28:12.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-08-31T22:28:12.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:28:12.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:28:12.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:12.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:28:12.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:28:12.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:28:12.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:12.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:28:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:28:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-08-31T22:28:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:28:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:28:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:28:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:28:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:28:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:28:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:28:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:28:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:28:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:28:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:28:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:28:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:28:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:28:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:28:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:28:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-08-31T22:28:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:28:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:28:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:28:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:28:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:28:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:28:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:28:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:28:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:28:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:28:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:28:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:28:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:28:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:28:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:28:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:28:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:12.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:12.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:12.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:12.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:12.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:12.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=158, tag_tid=23, entry_tid=6838], [object_number=157, tag_tid=23, entry_tid=6837], [object_number=156, tag_tid=23, entry_tid=6836], [object_number=159, tag_tid=23, entry_tid=6835]]], state=connected] 2024-08-31T22:28:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=158, tag_tid=23, entry_tid=6838], [object_number=157, tag_tid=23, entry_tid=6837], [object_number=156, tag_tid=23, entry_tid=6836], [object_number=159, tag_tid=23, entry_tid=6835]]], state=connected] 2024-08-31T22:28:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=158, tag_tid=23, entry_tid=6838' 2024-08-31T22:28:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=158, tag_tid=23, entry_tid=6838' 2024-08-31T22:28:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=158, tag_tid=23, entry_tid=6838' '!=' '' 2024-08-31T22:28:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=158, tag_tid=23, entry_tid=6838' = '' 2024-08-31T22:28:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=158, tag_tid=23, entry_tid=6838' 2024-08-31T22:28:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=158, tag_tid=23, entry_tid=6838' 2024-08-31T22:28:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:12.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:12.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:12.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:12.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=158, tag_tid=23, entry_tid=6838' 2024-08-31T22:28:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6838 2024-08-31T22:28:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 6838 2024-08-31T22:28:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=158, tag_tid=23, entry_tid=6938], [object_number=157, tag_tid=23, entry_tid=6937], [object_number=156, tag_tid=23, entry_tid=6936], [object_number=159, tag_tid=23, entry_tid=6935]]], state=connected] 2024-08-31T22:28:13.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=158, tag_tid=23, entry_tid=6938], [object_number=157, tag_tid=23, entry_tid=6937], [object_number=156, tag_tid=23, entry_tid=6936], [object_number=159, tag_tid=23, entry_tid=6935]]], state=connected] 2024-08-31T22:28:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=158, tag_tid=23, entry_tid=6938' 2024-08-31T22:28:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=158, tag_tid=23, entry_tid=6938' 2024-08-31T22:28:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=158, tag_tid=23, entry_tid=6938' '!=' 'object_number=158, tag_tid=23, entry_tid=6838' 2024-08-31T22:28:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=158, tag_tid=23, entry_tid=6938' = 'object_number=158, tag_tid=23, entry_tid=6838' 2024-08-31T22:28:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=158, tag_tid=23, entry_tid=6938' 2024-08-31T22:28:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=158, tag_tid=23, entry_tid=6938' 2024-08-31T22:28:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=158, tag_tid=23, entry_tid=6938' 2024-08-31T22:28:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6938 2024-08-31T22:28:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 6938 2024-08-31T22:28:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:13.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:13.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:13.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:13.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:13.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:13.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=156, tag_tid=23, entry_tid=7024], [object_number=159, tag_tid=23, entry_tid=7023], [object_number=158, tag_tid=23, entry_tid=7022], [object_number=157, tag_tid=23, entry_tid=7021]]], state=connected] 2024-08-31T22:28:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=156, tag_tid=23, entry_tid=7024], [object_number=159, tag_tid=23, entry_tid=7023], [object_number=158, tag_tid=23, entry_tid=7022], [object_number=157, tag_tid=23, entry_tid=7021]]], state=connected] 2024-08-31T22:28:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=156, tag_tid=23, entry_tid=7024' 2024-08-31T22:28:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=156, tag_tid=23, entry_tid=7024' 2024-08-31T22:28:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=156, tag_tid=23, entry_tid=7024' '!=' 'object_number=158, tag_tid=23, entry_tid=6938' 2024-08-31T22:28:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=156, tag_tid=23, entry_tid=7024' = 'object_number=158, tag_tid=23, entry_tid=6938' 2024-08-31T22:28:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=156, tag_tid=23, entry_tid=7024' 2024-08-31T22:28:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=156, tag_tid=23, entry_tid=7024' 2024-08-31T22:28:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:13.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:13.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:13.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:13.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=156, tag_tid=23, entry_tid=7024' 2024-08-31T22:28:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7024 2024-08-31T22:28:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 7024 2024-08-31T22:28:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=157, tag_tid=23, entry_tid=7109], [object_number=156, tag_tid=23, entry_tid=7108], [object_number=159, tag_tid=23, entry_tid=7107], [object_number=158, tag_tid=23, entry_tid=7106]]], state=connected] 2024-08-31T22:28:14.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=157, tag_tid=23, entry_tid=7109], [object_number=156, tag_tid=23, entry_tid=7108], [object_number=159, tag_tid=23, entry_tid=7107], [object_number=158, tag_tid=23, entry_tid=7106]]], state=connected] 2024-08-31T22:28:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=157, tag_tid=23, entry_tid=7109' 2024-08-31T22:28:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=157, tag_tid=23, entry_tid=7109' 2024-08-31T22:28:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=157, tag_tid=23, entry_tid=7109' '!=' 'object_number=156, tag_tid=23, entry_tid=7024' 2024-08-31T22:28:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=157, tag_tid=23, entry_tid=7109' = 'object_number=156, tag_tid=23, entry_tid=7024' 2024-08-31T22:28:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=157, tag_tid=23, entry_tid=7109' 2024-08-31T22:28:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=157, tag_tid=23, entry_tid=7109' 2024-08-31T22:28:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:14.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:14.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=157, tag_tid=23, entry_tid=7109' 2024-08-31T22:28:14.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:14.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7109 2024-08-31T22:28:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 7109 2024-08-31T22:28:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=156, tag_tid=23, entry_tid=7192], [object_number=159, tag_tid=23, entry_tid=7191], [object_number=158, tag_tid=23, entry_tid=7190], [object_number=157, tag_tid=23, entry_tid=7189]]], state=connected] 2024-08-31T22:28:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=156, tag_tid=23, entry_tid=7192], [object_number=159, tag_tid=23, entry_tid=7191], [object_number=158, tag_tid=23, entry_tid=7190], [object_number=157, tag_tid=23, entry_tid=7189]]], state=connected] 2024-08-31T22:28:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=156, tag_tid=23, entry_tid=7192' 2024-08-31T22:28:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=156, tag_tid=23, entry_tid=7192' 2024-08-31T22:28:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=156, tag_tid=23, entry_tid=7192' '!=' 'object_number=157, tag_tid=23, entry_tid=7109' 2024-08-31T22:28:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=156, tag_tid=23, entry_tid=7192' = 'object_number=157, tag_tid=23, entry_tid=7109' 2024-08-31T22:28:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=156, tag_tid=23, entry_tid=7192' 2024-08-31T22:28:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=156, tag_tid=23, entry_tid=7192' 2024-08-31T22:28:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=156, tag_tid=23, entry_tid=7192' 2024-08-31T22:28:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7192 2024-08-31T22:28:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 7192 2024-08-31T22:28:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=159, tag_tid=23, entry_tid=7279], [object_number=158, tag_tid=23, entry_tid=7278], [object_number=157, tag_tid=23, entry_tid=7277], [object_number=156, tag_tid=23, entry_tid=7276]]], state=connected] 2024-08-31T22:28:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:15.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=159, tag_tid=23, entry_tid=7279], [object_number=158, tag_tid=23, entry_tid=7278], [object_number=157, tag_tid=23, entry_tid=7277], [object_number=156, tag_tid=23, entry_tid=7276]]], state=connected] 2024-08-31T22:28:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=159, tag_tid=23, entry_tid=7279' 2024-08-31T22:28:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=159, tag_tid=23, entry_tid=7279' 2024-08-31T22:28:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=159, tag_tid=23, entry_tid=7279' '!=' 'object_number=156, tag_tid=23, entry_tid=7192' 2024-08-31T22:28:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=159, tag_tid=23, entry_tid=7279' = 'object_number=156, tag_tid=23, entry_tid=7192' 2024-08-31T22:28:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=159, tag_tid=23, entry_tid=7279' 2024-08-31T22:28:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:15.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:15.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:15.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=159, tag_tid=23, entry_tid=7279' 2024-08-31T22:28:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=159, tag_tid=23, entry_tid=7279' 2024-08-31T22:28:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7279 2024-08-31T22:28:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 7279 2024-08-31T22:28:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:15.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:15.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:15.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:15.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:15.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:15.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=156, tag_tid=23, entry_tid=7380], [object_number=159, tag_tid=23, entry_tid=7379], [object_number=158, tag_tid=23, entry_tid=7378], [object_number=157, tag_tid=23, entry_tid=7377]]], state=connected] 2024-08-31T22:28:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=156, tag_tid=23, entry_tid=7380], [object_number=159, tag_tid=23, entry_tid=7379], [object_number=158, tag_tid=23, entry_tid=7378], [object_number=157, tag_tid=23, entry_tid=7377]]], state=connected] 2024-08-31T22:28:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=156, tag_tid=23, entry_tid=7380' 2024-08-31T22:28:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=156, tag_tid=23, entry_tid=7380' 2024-08-31T22:28:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=156, tag_tid=23, entry_tid=7380' '!=' 'object_number=159, tag_tid=23, entry_tid=7279' 2024-08-31T22:28:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=156, tag_tid=23, entry_tid=7380' = 'object_number=159, tag_tid=23, entry_tid=7279' 2024-08-31T22:28:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=156, tag_tid=23, entry_tid=7380' 2024-08-31T22:28:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=156, tag_tid=23, entry_tid=7380' 2024-08-31T22:28:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=156, tag_tid=23, entry_tid=7380' 2024-08-31T22:28:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7380 2024-08-31T22:28:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 7380 2024-08-31T22:28:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:16.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:16.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:16.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:16.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:16.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=157, tag_tid=23, entry_tid=7461], [object_number=156, tag_tid=23, entry_tid=7460], [object_number=159, tag_tid=23, entry_tid=7459], [object_number=158, tag_tid=23, entry_tid=7458]]], state=connected] 2024-08-31T22:28:16.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=157, tag_tid=23, entry_tid=7461], [object_number=156, tag_tid=23, entry_tid=7460], [object_number=159, tag_tid=23, entry_tid=7459], [object_number=158, tag_tid=23, entry_tid=7458]]], state=connected] 2024-08-31T22:28:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=157, tag_tid=23, entry_tid=7461' 2024-08-31T22:28:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=157, tag_tid=23, entry_tid=7461' 2024-08-31T22:28:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=157, tag_tid=23, entry_tid=7461' '!=' 'object_number=156, tag_tid=23, entry_tid=7380' 2024-08-31T22:28:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=157, tag_tid=23, entry_tid=7461' = 'object_number=156, tag_tid=23, entry_tid=7380' 2024-08-31T22:28:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=157, tag_tid=23, entry_tid=7461' 2024-08-31T22:28:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=157, tag_tid=23, entry_tid=7461' 2024-08-31T22:28:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=157, tag_tid=23, entry_tid=7461' 2024-08-31T22:28:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7461 2024-08-31T22:28:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 7461 2024-08-31T22:28:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:16.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:16.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:16.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:16.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:16.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=157, tag_tid=23, entry_tid=7541], [object_number=156, tag_tid=23, entry_tid=7540], [object_number=159, tag_tid=23, entry_tid=7539], [object_number=158, tag_tid=23, entry_tid=7538]]], state=connected] 2024-08-31T22:28:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:16.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=157, tag_tid=23, entry_tid=7541], [object_number=156, tag_tid=23, entry_tid=7540], [object_number=159, tag_tid=23, entry_tid=7539], [object_number=158, tag_tid=23, entry_tid=7538]]], state=connected] 2024-08-31T22:28:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=157, tag_tid=23, entry_tid=7541' 2024-08-31T22:28:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=157, tag_tid=23, entry_tid=7541' 2024-08-31T22:28:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=157, tag_tid=23, entry_tid=7541' '!=' 'object_number=157, tag_tid=23, entry_tid=7461' 2024-08-31T22:28:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=157, tag_tid=23, entry_tid=7541' = 'object_number=157, tag_tid=23, entry_tid=7461' 2024-08-31T22:28:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=157, tag_tid=23, entry_tid=7541' 2024-08-31T22:28:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=157, tag_tid=23, entry_tid=7541' 2024-08-31T22:28:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:16.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=157, tag_tid=23, entry_tid=7541' 2024-08-31T22:28:16.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:16.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:16.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7541 2024-08-31T22:28:16.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 7541 2024-08-31T22:28:16.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:16.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:16.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=159, tag_tid=23, entry_tid=7619], [object_number=158, tag_tid=23, entry_tid=7618], [object_number=157, tag_tid=23, entry_tid=7617], [object_number=156, tag_tid=23, entry_tid=7616]]], state=connected] 2024-08-31T22:28:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=159, tag_tid=23, entry_tid=7619], [object_number=158, tag_tid=23, entry_tid=7618], [object_number=157, tag_tid=23, entry_tid=7617], [object_number=156, tag_tid=23, entry_tid=7616]]], state=connected] 2024-08-31T22:28:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=159, tag_tid=23, entry_tid=7619' 2024-08-31T22:28:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=159, tag_tid=23, entry_tid=7619' 2024-08-31T22:28:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=159, tag_tid=23, entry_tid=7619' '!=' 'object_number=157, tag_tid=23, entry_tid=7541' 2024-08-31T22:28:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=159, tag_tid=23, entry_tid=7619' = 'object_number=157, tag_tid=23, entry_tid=7541' 2024-08-31T22:28:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=159, tag_tid=23, entry_tid=7619' 2024-08-31T22:28:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:17.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=159, tag_tid=23, entry_tid=7619' 2024-08-31T22:28:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=159, tag_tid=23, entry_tid=7619' 2024-08-31T22:28:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7619 2024-08-31T22:28:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 7619 2024-08-31T22:28:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:17.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:17.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:17.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:17.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:17.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=157, tag_tid=23, entry_tid=7701], [object_number=156, tag_tid=23, entry_tid=7700], [object_number=159, tag_tid=23, entry_tid=7699], [object_number=158, tag_tid=23, entry_tid=7698]]], state=connected] 2024-08-31T22:28:17.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=157, tag_tid=23, entry_tid=7701], [object_number=156, tag_tid=23, entry_tid=7700], [object_number=159, tag_tid=23, entry_tid=7699], [object_number=158, tag_tid=23, entry_tid=7698]]], state=connected] 2024-08-31T22:28:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=157, tag_tid=23, entry_tid=7701' 2024-08-31T22:28:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=157, tag_tid=23, entry_tid=7701' 2024-08-31T22:28:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=157, tag_tid=23, entry_tid=7701' '!=' 'object_number=159, tag_tid=23, entry_tid=7619' 2024-08-31T22:28:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=157, tag_tid=23, entry_tid=7701' = 'object_number=159, tag_tid=23, entry_tid=7619' 2024-08-31T22:28:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=157, tag_tid=23, entry_tid=7701' 2024-08-31T22:28:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=157, tag_tid=23, entry_tid=7701' 2024-08-31T22:28:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=157, tag_tid=23, entry_tid=7701' 2024-08-31T22:28:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7701 2024-08-31T22:28:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 7701 2024-08-31T22:28:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:18.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:18.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:18.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:18.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:18.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:18.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:18.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:18.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:18.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:18.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:18.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:18.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:18.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:18.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:18.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:18.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:18.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:18.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:18.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:18.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=156, tag_tid=23, entry_tid=7780], [object_number=159, tag_tid=23, entry_tid=7779], [object_number=158, tag_tid=23, entry_tid=7778], [object_number=157, tag_tid=23, entry_tid=7777]]], state=connected] 2024-08-31T22:28:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:18.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:18.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:18.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=156, tag_tid=23, entry_tid=7780], [object_number=159, tag_tid=23, entry_tid=7779], [object_number=158, tag_tid=23, entry_tid=7778], [object_number=157, tag_tid=23, entry_tid=7777]]], state=connected] 2024-08-31T22:28:18.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=156, tag_tid=23, entry_tid=7780' 2024-08-31T22:28:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=156, tag_tid=23, entry_tid=7780' 2024-08-31T22:28:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=156, tag_tid=23, entry_tid=7780' '!=' 'object_number=157, tag_tid=23, entry_tid=7701' 2024-08-31T22:28:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=156, tag_tid=23, entry_tid=7780' = 'object_number=157, tag_tid=23, entry_tid=7701' 2024-08-31T22:28:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=156, tag_tid=23, entry_tid=7780' 2024-08-31T22:28:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=156, tag_tid=23, entry_tid=7780' 2024-08-31T22:28:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=156, tag_tid=23, entry_tid=7780' 2024-08-31T22:28:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7780 2024-08-31T22:28:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 7780 2024-08-31T22:28:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=159, tag_tid=23, entry_tid=7887], [object_number=158, tag_tid=23, entry_tid=7886], [object_number=157, tag_tid=23, entry_tid=7885], [object_number=156, tag_tid=23, entry_tid=7884]]], state=connected] 2024-08-31T22:28:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=159, tag_tid=23, entry_tid=7887], [object_number=158, tag_tid=23, entry_tid=7886], [object_number=157, tag_tid=23, entry_tid=7885], [object_number=156, tag_tid=23, entry_tid=7884]]], state=connected] 2024-08-31T22:28:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=159, tag_tid=23, entry_tid=7887' 2024-08-31T22:28:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=159, tag_tid=23, entry_tid=7887' 2024-08-31T22:28:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=159, tag_tid=23, entry_tid=7887' '!=' 'object_number=156, tag_tid=23, entry_tid=7780' 2024-08-31T22:28:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=159, tag_tid=23, entry_tid=7887' = 'object_number=156, tag_tid=23, entry_tid=7780' 2024-08-31T22:28:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=159, tag_tid=23, entry_tid=7887' 2024-08-31T22:28:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=159, tag_tid=23, entry_tid=7887' 2024-08-31T22:28:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=159, tag_tid=23, entry_tid=7887' 2024-08-31T22:28:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7887 2024-08-31T22:28:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 7887 2024-08-31T22:28:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=158, tag_tid=23, entry_tid=7978], [object_number=157, tag_tid=23, entry_tid=7977], [object_number=156, tag_tid=23, entry_tid=7976], [object_number=159, tag_tid=23, entry_tid=7975]]], state=connected] 2024-08-31T22:28:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:19.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:19.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=158, tag_tid=23, entry_tid=7978], [object_number=157, tag_tid=23, entry_tid=7977], [object_number=156, tag_tid=23, entry_tid=7976], [object_number=159, tag_tid=23, entry_tid=7975]]], state=connected] 2024-08-31T22:28:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=158, tag_tid=23, entry_tid=7978' 2024-08-31T22:28:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=158, tag_tid=23, entry_tid=7978' 2024-08-31T22:28:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=158, tag_tid=23, entry_tid=7978' '!=' 'object_number=159, tag_tid=23, entry_tid=7887' 2024-08-31T22:28:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=158, tag_tid=23, entry_tid=7978' = 'object_number=159, tag_tid=23, entry_tid=7887' 2024-08-31T22:28:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=158, tag_tid=23, entry_tid=7978' 2024-08-31T22:28:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:19.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:19.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=158, tag_tid=23, entry_tid=7978' 2024-08-31T22:28:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=158, tag_tid=23, entry_tid=7978' 2024-08-31T22:28:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7978 2024-08-31T22:28:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 7978 2024-08-31T22:28:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:19.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=159, tag_tid=23, entry_tid=8067], [object_number=158, tag_tid=23, entry_tid=8066], [object_number=157, tag_tid=23, entry_tid=8065], [object_number=156, tag_tid=23, entry_tid=8064]]], state=connected] 2024-08-31T22:28:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=159, tag_tid=23, entry_tid=8067], [object_number=158, tag_tid=23, entry_tid=8066], [object_number=157, tag_tid=23, entry_tid=8065], [object_number=156, tag_tid=23, entry_tid=8064]]], state=connected] 2024-08-31T22:28:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=159, tag_tid=23, entry_tid=8067' 2024-08-31T22:28:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=159, tag_tid=23, entry_tid=8067' 2024-08-31T22:28:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=159, tag_tid=23, entry_tid=8067' '!=' 'object_number=158, tag_tid=23, entry_tid=7978' 2024-08-31T22:28:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=159, tag_tid=23, entry_tid=8067' = 'object_number=158, tag_tid=23, entry_tid=7978' 2024-08-31T22:28:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=159, tag_tid=23, entry_tid=8067' 2024-08-31T22:28:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=159, tag_tid=23, entry_tid=8067' 2024-08-31T22:28:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=159, tag_tid=23, entry_tid=8067' 2024-08-31T22:28:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8067 2024-08-31T22:28:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 8067 2024-08-31T22:28:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=156, tag_tid=23, entry_tid=8160], [object_number=159, tag_tid=23, entry_tid=8159], [object_number=158, tag_tid=23, entry_tid=8158], [object_number=157, tag_tid=23, entry_tid=8157]]], state=connected] 2024-08-31T22:28:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=156, tag_tid=23, entry_tid=8160], [object_number=159, tag_tid=23, entry_tid=8159], [object_number=158, tag_tid=23, entry_tid=8158], [object_number=157, tag_tid=23, entry_tid=8157]]], state=connected] 2024-08-31T22:28:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=156, tag_tid=23, entry_tid=8160' 2024-08-31T22:28:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=156, tag_tid=23, entry_tid=8160' 2024-08-31T22:28:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=156, tag_tid=23, entry_tid=8160' '!=' 'object_number=159, tag_tid=23, entry_tid=8067' 2024-08-31T22:28:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=156, tag_tid=23, entry_tid=8160' = 'object_number=159, tag_tid=23, entry_tid=8067' 2024-08-31T22:28:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=156, tag_tid=23, entry_tid=8160' 2024-08-31T22:28:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=156, tag_tid=23, entry_tid=8160' 2024-08-31T22:28:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:20.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:20.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=156, tag_tid=23, entry_tid=8160' 2024-08-31T22:28:20.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:20.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8160 2024-08-31T22:28:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 8160 2024-08-31T22:28:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:20.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=158, tag_tid=23, entry_tid=8258], [object_number=157, tag_tid=23, entry_tid=8257], [object_number=156, tag_tid=23, entry_tid=8256], [object_number=159, tag_tid=23, entry_tid=8255]]], state=connected] 2024-08-31T22:28:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=158, tag_tid=23, entry_tid=8258], [object_number=157, tag_tid=23, entry_tid=8257], [object_number=156, tag_tid=23, entry_tid=8256], [object_number=159, tag_tid=23, entry_tid=8255]]], state=connected] 2024-08-31T22:28:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=158, tag_tid=23, entry_tid=8258' 2024-08-31T22:28:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=158, tag_tid=23, entry_tid=8258' 2024-08-31T22:28:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=158, tag_tid=23, entry_tid=8258' '!=' 'object_number=156, tag_tid=23, entry_tid=8160' 2024-08-31T22:28:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=158, tag_tid=23, entry_tid=8258' = 'object_number=156, tag_tid=23, entry_tid=8160' 2024-08-31T22:28:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=158, tag_tid=23, entry_tid=8258' 2024-08-31T22:28:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=158, tag_tid=23, entry_tid=8258' 2024-08-31T22:28:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=158, tag_tid=23, entry_tid=8258' 2024-08-31T22:28:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8258 2024-08-31T22:28:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 8258 2024-08-31T22:28:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:21.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:21.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:21.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:21.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=156, tag_tid=23, entry_tid=8364], [object_number=159, tag_tid=23, entry_tid=8363], [object_number=158, tag_tid=23, entry_tid=8362], [object_number=157, tag_tid=23, entry_tid=8361]]], state=connected] 2024-08-31T22:28:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=156, tag_tid=23, entry_tid=8364], [object_number=159, tag_tid=23, entry_tid=8363], [object_number=158, tag_tid=23, entry_tid=8362], [object_number=157, tag_tid=23, entry_tid=8361]]], state=connected] 2024-08-31T22:28:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=156, tag_tid=23, entry_tid=8364' 2024-08-31T22:28:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=156, tag_tid=23, entry_tid=8364' 2024-08-31T22:28:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=156, tag_tid=23, entry_tid=8364' '!=' 'object_number=158, tag_tid=23, entry_tid=8258' 2024-08-31T22:28:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=156, tag_tid=23, entry_tid=8364' = 'object_number=158, tag_tid=23, entry_tid=8258' 2024-08-31T22:28:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=156, tag_tid=23, entry_tid=8364' 2024-08-31T22:28:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=156, tag_tid=23, entry_tid=8364' 2024-08-31T22:28:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:21.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=156, tag_tid=23, entry_tid=8364' 2024-08-31T22:28:21.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:21.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8364 2024-08-31T22:28:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 8364 2024-08-31T22:28:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=157, tag_tid=23, entry_tid=8461], [object_number=156, tag_tid=23, entry_tid=8460], [object_number=159, tag_tid=23, entry_tid=8459], [object_number=158, tag_tid=23, entry_tid=8458]]], state=connected] 2024-08-31T22:28:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:22.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:22.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:22.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=157, tag_tid=23, entry_tid=8461], [object_number=156, tag_tid=23, entry_tid=8460], [object_number=159, tag_tid=23, entry_tid=8459], [object_number=158, tag_tid=23, entry_tid=8458]]], state=connected] 2024-08-31T22:28:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=157, tag_tid=23, entry_tid=8461' 2024-08-31T22:28:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=157, tag_tid=23, entry_tid=8461' 2024-08-31T22:28:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=157, tag_tid=23, entry_tid=8461' '!=' 'object_number=156, tag_tid=23, entry_tid=8364' 2024-08-31T22:28:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=157, tag_tid=23, entry_tid=8461' = 'object_number=156, tag_tid=23, entry_tid=8364' 2024-08-31T22:28:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=157, tag_tid=23, entry_tid=8461' 2024-08-31T22:28:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=157, tag_tid=23, entry_tid=8461' 2024-08-31T22:28:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=157, tag_tid=23, entry_tid=8461' 2024-08-31T22:28:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8461 2024-08-31T22:28:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 8461 2024-08-31T22:28:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:22.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:22.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:22.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:22.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=160, tag_tid=23, entry_tid=8552], [object_number=159, tag_tid=23, entry_tid=8551], [object_number=158, tag_tid=23, entry_tid=8550], [object_number=157, tag_tid=23, entry_tid=8549]]], state=connected] 2024-08-31T22:28:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=160, tag_tid=23, entry_tid=8552], [object_number=159, tag_tid=23, entry_tid=8551], [object_number=158, tag_tid=23, entry_tid=8550], [object_number=157, tag_tid=23, entry_tid=8549]]], state=connected] 2024-08-31T22:28:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=160, tag_tid=23, entry_tid=8552' 2024-08-31T22:28:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=160, tag_tid=23, entry_tid=8552' 2024-08-31T22:28:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=160, tag_tid=23, entry_tid=8552' '!=' 'object_number=157, tag_tid=23, entry_tid=8461' 2024-08-31T22:28:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=160, tag_tid=23, entry_tid=8552' = 'object_number=157, tag_tid=23, entry_tid=8461' 2024-08-31T22:28:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=160, tag_tid=23, entry_tid=8552' 2024-08-31T22:28:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=160, tag_tid=23, entry_tid=8552' 2024-08-31T22:28:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=160, tag_tid=23, entry_tid=8552' 2024-08-31T22:28:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8552 2024-08-31T22:28:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 8552 2024-08-31T22:28:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:23.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=158, tag_tid=23, entry_tid=8610], [object_number=161, tag_tid=23, entry_tid=8609], [object_number=160, tag_tid=23, entry_tid=8608], [object_number=163, tag_tid=23, entry_tid=8607]]], state=connected] 2024-08-31T22:28:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=158, tag_tid=23, entry_tid=8610], [object_number=161, tag_tid=23, entry_tid=8609], [object_number=160, tag_tid=23, entry_tid=8608], [object_number=163, tag_tid=23, entry_tid=8607]]], state=connected] 2024-08-31T22:28:23.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=158, tag_tid=23, entry_tid=8610' 2024-08-31T22:28:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=158, tag_tid=23, entry_tid=8610' 2024-08-31T22:28:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=158, tag_tid=23, entry_tid=8610' '!=' 'object_number=160, tag_tid=23, entry_tid=8552' 2024-08-31T22:28:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=158, tag_tid=23, entry_tid=8610' = 'object_number=160, tag_tid=23, entry_tid=8552' 2024-08-31T22:28:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=158, tag_tid=23, entry_tid=8610' 2024-08-31T22:28:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=158, tag_tid=23, entry_tid=8610' 2024-08-31T22:28:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=158, tag_tid=23, entry_tid=8610' 2024-08-31T22:28:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:23.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8610 2024-08-31T22:28:23.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 8610 2024-08-31T22:28:23.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:23.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:23.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=8665], [object_number=160, tag_tid=23, entry_tid=8664], [object_number=163, tag_tid=23, entry_tid=8663], [object_number=158, tag_tid=23, entry_tid=8662]]], state=connected] 2024-08-31T22:28:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:23.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 39 2024-08-31T22:28:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=8665], [object_number=160, tag_tid=23, entry_tid=8664], [object_number=163, tag_tid=23, entry_tid=8663], [object_number=158, tag_tid=23, entry_tid=8662]]], state=connected] 2024-08-31T22:28:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=161, tag_tid=23, entry_tid=8665' 2024-08-31T22:28:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=161, tag_tid=23, entry_tid=8665' 2024-08-31T22:28:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=8665' '!=' 'object_number=158, tag_tid=23, entry_tid=8610' 2024-08-31T22:28:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=8665' = 'object_number=158, tag_tid=23, entry_tid=8610' 2024-08-31T22:28:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=161, tag_tid=23, entry_tid=8665' 2024-08-31T22:28:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=8665' 2024-08-31T22:28:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=8665' 2024-08-31T22:28:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8665 2024-08-31T22:28:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 8665 2024-08-31T22:28:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:24.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=8730], [object_number=161, tag_tid=23, entry_tid=8729], [object_number=160, tag_tid=23, entry_tid=8728], [object_number=163, tag_tid=23, entry_tid=8727]]], state=connected] 2024-08-31T22:28:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=8730], [object_number=161, tag_tid=23, entry_tid=8729], [object_number=160, tag_tid=23, entry_tid=8728], [object_number=163, tag_tid=23, entry_tid=8727]]], state=connected] 2024-08-31T22:28:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=162, tag_tid=23, entry_tid=8730' 2024-08-31T22:28:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=162, tag_tid=23, entry_tid=8730' 2024-08-31T22:28:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=8730' '!=' 'object_number=161, tag_tid=23, entry_tid=8665' 2024-08-31T22:28:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=8730' = 'object_number=161, tag_tid=23, entry_tid=8665' 2024-08-31T22:28:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=162, tag_tid=23, entry_tid=8730' 2024-08-31T22:28:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=8730' 2024-08-31T22:28:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=8730' 2024-08-31T22:28:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8730 2024-08-31T22:28:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 8730 2024-08-31T22:28:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:24.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:24.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=8821], [object_number=160, tag_tid=23, entry_tid=8820], [object_number=163, tag_tid=23, entry_tid=8819], [object_number=162, tag_tid=23, entry_tid=8818]]], state=connected] 2024-08-31T22:28:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=8821], [object_number=160, tag_tid=23, entry_tid=8820], [object_number=163, tag_tid=23, entry_tid=8819], [object_number=162, tag_tid=23, entry_tid=8818]]], state=connected] 2024-08-31T22:28:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=161, tag_tid=23, entry_tid=8821' 2024-08-31T22:28:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=161, tag_tid=23, entry_tid=8821' 2024-08-31T22:28:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=8821' '!=' 'object_number=162, tag_tid=23, entry_tid=8730' 2024-08-31T22:28:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=8821' = 'object_number=162, tag_tid=23, entry_tid=8730' 2024-08-31T22:28:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=161, tag_tid=23, entry_tid=8821' 2024-08-31T22:28:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=8821' 2024-08-31T22:28:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=8821' 2024-08-31T22:28:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8821 2024-08-31T22:28:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 8821 2024-08-31T22:28:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=163, tag_tid=23, entry_tid=8907], [object_number=162, tag_tid=23, entry_tid=8906], [object_number=161, tag_tid=23, entry_tid=8905], [object_number=160, tag_tid=23, entry_tid=8904]]], state=connected] 2024-08-31T22:28:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=163, tag_tid=23, entry_tid=8907], [object_number=162, tag_tid=23, entry_tid=8906], [object_number=161, tag_tid=23, entry_tid=8905], [object_number=160, tag_tid=23, entry_tid=8904]]], state=connected] 2024-08-31T22:28:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=163, tag_tid=23, entry_tid=8907' 2024-08-31T22:28:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=163, tag_tid=23, entry_tid=8907' 2024-08-31T22:28:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=163, tag_tid=23, entry_tid=8907' '!=' 'object_number=161, tag_tid=23, entry_tid=8821' 2024-08-31T22:28:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=163, tag_tid=23, entry_tid=8907' = 'object_number=161, tag_tid=23, entry_tid=8821' 2024-08-31T22:28:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=163, tag_tid=23, entry_tid=8907' 2024-08-31T22:28:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=163, tag_tid=23, entry_tid=8907' 2024-08-31T22:28:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=163, tag_tid=23, entry_tid=8907' 2024-08-31T22:28:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8907 2024-08-31T22:28:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 8907 2024-08-31T22:28:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:25.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=163, tag_tid=23, entry_tid=8995], [object_number=162, tag_tid=23, entry_tid=8994], [object_number=161, tag_tid=23, entry_tid=8993], [object_number=160, tag_tid=23, entry_tid=8992]]], state=connected] 2024-08-31T22:28:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=163, tag_tid=23, entry_tid=8995], [object_number=162, tag_tid=23, entry_tid=8994], [object_number=161, tag_tid=23, entry_tid=8993], [object_number=160, tag_tid=23, entry_tid=8992]]], state=connected] 2024-08-31T22:28:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=163, tag_tid=23, entry_tid=8995' 2024-08-31T22:28:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=163, tag_tid=23, entry_tid=8995' 2024-08-31T22:28:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=163, tag_tid=23, entry_tid=8995' '!=' 'object_number=163, tag_tid=23, entry_tid=8907' 2024-08-31T22:28:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=163, tag_tid=23, entry_tid=8995' = 'object_number=163, tag_tid=23, entry_tid=8907' 2024-08-31T22:28:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=163, tag_tid=23, entry_tid=8995' 2024-08-31T22:28:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=163, tag_tid=23, entry_tid=8995' 2024-08-31T22:28:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=163, tag_tid=23, entry_tid=8995' 2024-08-31T22:28:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8995 2024-08-31T22:28:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 8995 2024-08-31T22:28:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=9081], [object_number=160, tag_tid=23, entry_tid=9080], [object_number=163, tag_tid=23, entry_tid=9079], [object_number=162, tag_tid=23, entry_tid=9078]]], state=connected] 2024-08-31T22:28:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=9081], [object_number=160, tag_tid=23, entry_tid=9080], [object_number=163, tag_tid=23, entry_tid=9079], [object_number=162, tag_tid=23, entry_tid=9078]]], state=connected] 2024-08-31T22:28:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=161, tag_tid=23, entry_tid=9081' 2024-08-31T22:28:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=161, tag_tid=23, entry_tid=9081' 2024-08-31T22:28:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=9081' '!=' 'object_number=163, tag_tid=23, entry_tid=8995' 2024-08-31T22:28:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=9081' = 'object_number=163, tag_tid=23, entry_tid=8995' 2024-08-31T22:28:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=161, tag_tid=23, entry_tid=9081' 2024-08-31T22:28:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=9081' 2024-08-31T22:28:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=9081' 2024-08-31T22:28:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9081 2024-08-31T22:28:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 9081 2024-08-31T22:28:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:26.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:26.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=163, tag_tid=23, entry_tid=9163], [object_number=162, tag_tid=23, entry_tid=9162], [object_number=161, tag_tid=23, entry_tid=9161], [object_number=160, tag_tid=23, entry_tid=9160]]], state=connected] 2024-08-31T22:28:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=163, tag_tid=23, entry_tid=9163], [object_number=162, tag_tid=23, entry_tid=9162], [object_number=161, tag_tid=23, entry_tid=9161], [object_number=160, tag_tid=23, entry_tid=9160]]], state=connected] 2024-08-31T22:28:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=163, tag_tid=23, entry_tid=9163' 2024-08-31T22:28:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=163, tag_tid=23, entry_tid=9163' 2024-08-31T22:28:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=163, tag_tid=23, entry_tid=9163' '!=' 'object_number=161, tag_tid=23, entry_tid=9081' 2024-08-31T22:28:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=163, tag_tid=23, entry_tid=9163' = 'object_number=161, tag_tid=23, entry_tid=9081' 2024-08-31T22:28:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=163, tag_tid=23, entry_tid=9163' 2024-08-31T22:28:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=163, tag_tid=23, entry_tid=9163' 2024-08-31T22:28:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=163, tag_tid=23, entry_tid=9163' 2024-08-31T22:28:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9163 2024-08-31T22:28:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 9163 2024-08-31T22:28:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=163, tag_tid=23, entry_tid=9247], [object_number=162, tag_tid=23, entry_tid=9246], [object_number=161, tag_tid=23, entry_tid=9245], [object_number=160, tag_tid=23, entry_tid=9244]]], state=connected] 2024-08-31T22:28:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:27.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:27.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:27.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:27.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:27.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:27.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=163, tag_tid=23, entry_tid=9247], [object_number=162, tag_tid=23, entry_tid=9246], [object_number=161, tag_tid=23, entry_tid=9245], [object_number=160, tag_tid=23, entry_tid=9244]]], state=connected] 2024-08-31T22:28:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=163, tag_tid=23, entry_tid=9247' 2024-08-31T22:28:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=163, tag_tid=23, entry_tid=9247' 2024-08-31T22:28:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=163, tag_tid=23, entry_tid=9247' '!=' 'object_number=163, tag_tid=23, entry_tid=9163' 2024-08-31T22:28:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=163, tag_tid=23, entry_tid=9247' = 'object_number=163, tag_tid=23, entry_tid=9163' 2024-08-31T22:28:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=163, tag_tid=23, entry_tid=9247' 2024-08-31T22:28:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=163, tag_tid=23, entry_tid=9247' 2024-08-31T22:28:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=163, tag_tid=23, entry_tid=9247' 2024-08-31T22:28:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9247 2024-08-31T22:28:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 9247 2024-08-31T22:28:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=9333], [object_number=160, tag_tid=23, entry_tid=9332], [object_number=163, tag_tid=23, entry_tid=9331], [object_number=162, tag_tid=23, entry_tid=9330]]], state=connected] 2024-08-31T22:28:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:27.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:27.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=9333], [object_number=160, tag_tid=23, entry_tid=9332], [object_number=163, tag_tid=23, entry_tid=9331], [object_number=162, tag_tid=23, entry_tid=9330]]], state=connected] 2024-08-31T22:28:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=161, tag_tid=23, entry_tid=9333' 2024-08-31T22:28:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=161, tag_tid=23, entry_tid=9333' 2024-08-31T22:28:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=9333' '!=' 'object_number=163, tag_tid=23, entry_tid=9247' 2024-08-31T22:28:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=9333' = 'object_number=163, tag_tid=23, entry_tid=9247' 2024-08-31T22:28:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=161, tag_tid=23, entry_tid=9333' 2024-08-31T22:28:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=9333' 2024-08-31T22:28:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=9333' 2024-08-31T22:28:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9333 2024-08-31T22:28:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 9333 2024-08-31T22:28:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:28.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:28.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:28.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:28.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:28.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:28.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:28.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:28.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=9417], [object_number=160, tag_tid=23, entry_tid=9416], [object_number=163, tag_tid=23, entry_tid=9415], [object_number=162, tag_tid=23, entry_tid=9414]]], state=connected] 2024-08-31T22:28:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:28.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:28.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:28.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:28.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:28.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=9417], [object_number=160, tag_tid=23, entry_tid=9416], [object_number=163, tag_tid=23, entry_tid=9415], [object_number=162, tag_tid=23, entry_tid=9414]]], state=connected] 2024-08-31T22:28:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=161, tag_tid=23, entry_tid=9417' 2024-08-31T22:28:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=161, tag_tid=23, entry_tid=9417' 2024-08-31T22:28:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=9417' '!=' 'object_number=161, tag_tid=23, entry_tid=9333' 2024-08-31T22:28:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=9417' = 'object_number=161, tag_tid=23, entry_tid=9333' 2024-08-31T22:28:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=161, tag_tid=23, entry_tid=9417' 2024-08-31T22:28:28.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:28.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:28.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:28.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=9417' 2024-08-31T22:28:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=9417' 2024-08-31T22:28:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9417 2024-08-31T22:28:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 9417 2024-08-31T22:28:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:28.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:28.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:28.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=9493], [object_number=160, tag_tid=23, entry_tid=9492], [object_number=163, tag_tid=23, entry_tid=9491], [object_number=162, tag_tid=23, entry_tid=9490]]], state=connected] 2024-08-31T22:28:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=9493], [object_number=160, tag_tid=23, entry_tid=9492], [object_number=163, tag_tid=23, entry_tid=9491], [object_number=162, tag_tid=23, entry_tid=9490]]], state=connected] 2024-08-31T22:28:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=161, tag_tid=23, entry_tid=9493' 2024-08-31T22:28:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=161, tag_tid=23, entry_tid=9493' 2024-08-31T22:28:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=9493' '!=' 'object_number=161, tag_tid=23, entry_tid=9417' 2024-08-31T22:28:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=9493' = 'object_number=161, tag_tid=23, entry_tid=9417' 2024-08-31T22:28:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=161, tag_tid=23, entry_tid=9493' 2024-08-31T22:28:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=9493' 2024-08-31T22:28:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=9493' 2024-08-31T22:28:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9493 2024-08-31T22:28:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 9493 2024-08-31T22:28:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=9610], [object_number=161, tag_tid=23, entry_tid=9609], [object_number=160, tag_tid=23, entry_tid=9608], [object_number=163, tag_tid=23, entry_tid=9607]]], state=connected] 2024-08-31T22:28:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:29.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:29.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:29.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:29.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:29.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:29.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:29.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:29.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:29.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:29.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:29.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=9610], [object_number=161, tag_tid=23, entry_tid=9609], [object_number=160, tag_tid=23, entry_tid=9608], [object_number=163, tag_tid=23, entry_tid=9607]]], state=connected] 2024-08-31T22:28:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=162, tag_tid=23, entry_tid=9610' 2024-08-31T22:28:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=162, tag_tid=23, entry_tid=9610' 2024-08-31T22:28:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=9610' '!=' 'object_number=161, tag_tid=23, entry_tid=9493' 2024-08-31T22:28:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=9610' = 'object_number=161, tag_tid=23, entry_tid=9493' 2024-08-31T22:28:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=162, tag_tid=23, entry_tid=9610' 2024-08-31T22:28:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=9610' 2024-08-31T22:28:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=9610' 2024-08-31T22:28:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9610 2024-08-31T22:28:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 9610 2024-08-31T22:28:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:29.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:29.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:29.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=160, tag_tid=23, entry_tid=9692], [object_number=163, tag_tid=23, entry_tid=9691], [object_number=162, tag_tid=23, entry_tid=9690], [object_number=161, tag_tid=23, entry_tid=9689]]], state=connected] 2024-08-31T22:28:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=160, tag_tid=23, entry_tid=9692], [object_number=163, tag_tid=23, entry_tid=9691], [object_number=162, tag_tid=23, entry_tid=9690], [object_number=161, tag_tid=23, entry_tid=9689]]], state=connected] 2024-08-31T22:28:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=160, tag_tid=23, entry_tid=9692' 2024-08-31T22:28:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=160, tag_tid=23, entry_tid=9692' 2024-08-31T22:28:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=160, tag_tid=23, entry_tid=9692' '!=' 'object_number=162, tag_tid=23, entry_tid=9610' 2024-08-31T22:28:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=160, tag_tid=23, entry_tid=9692' = 'object_number=162, tag_tid=23, entry_tid=9610' 2024-08-31T22:28:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=160, tag_tid=23, entry_tid=9692' 2024-08-31T22:28:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=160, tag_tid=23, entry_tid=9692' 2024-08-31T22:28:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=160, tag_tid=23, entry_tid=9692' 2024-08-31T22:28:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9692 2024-08-31T22:28:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 9692 2024-08-31T22:28:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=160, tag_tid=23, entry_tid=9772], [object_number=163, tag_tid=23, entry_tid=9771], [object_number=162, tag_tid=23, entry_tid=9770], [object_number=161, tag_tid=23, entry_tid=9769]]], state=connected] 2024-08-31T22:28:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=160, tag_tid=23, entry_tid=9772], [object_number=163, tag_tid=23, entry_tid=9771], [object_number=162, tag_tid=23, entry_tid=9770], [object_number=161, tag_tid=23, entry_tid=9769]]], state=connected] 2024-08-31T22:28:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=160, tag_tid=23, entry_tid=9772' 2024-08-31T22:28:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=160, tag_tid=23, entry_tid=9772' 2024-08-31T22:28:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=160, tag_tid=23, entry_tid=9772' '!=' 'object_number=160, tag_tid=23, entry_tid=9692' 2024-08-31T22:28:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=160, tag_tid=23, entry_tid=9772' = 'object_number=160, tag_tid=23, entry_tid=9692' 2024-08-31T22:28:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=160, tag_tid=23, entry_tid=9772' 2024-08-31T22:28:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=160, tag_tid=23, entry_tid=9772' 2024-08-31T22:28:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:30.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=160, tag_tid=23, entry_tid=9772' 2024-08-31T22:28:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9772 2024-08-31T22:28:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 9772 2024-08-31T22:28:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:30.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:30.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:30.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:30.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=163, tag_tid=23, entry_tid=9851], [object_number=162, tag_tid=23, entry_tid=9850], [object_number=161, tag_tid=23, entry_tid=9849], [object_number=160, tag_tid=23, entry_tid=9848]]], state=connected] 2024-08-31T22:28:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=163, tag_tid=23, entry_tid=9851], [object_number=162, tag_tid=23, entry_tid=9850], [object_number=161, tag_tid=23, entry_tid=9849], [object_number=160, tag_tid=23, entry_tid=9848]]], state=connected] 2024-08-31T22:28:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=163, tag_tid=23, entry_tid=9851' 2024-08-31T22:28:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=163, tag_tid=23, entry_tid=9851' 2024-08-31T22:28:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=163, tag_tid=23, entry_tid=9851' '!=' 'object_number=160, tag_tid=23, entry_tid=9772' 2024-08-31T22:28:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=163, tag_tid=23, entry_tid=9851' = 'object_number=160, tag_tid=23, entry_tid=9772' 2024-08-31T22:28:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=163, tag_tid=23, entry_tid=9851' 2024-08-31T22:28:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=163, tag_tid=23, entry_tid=9851' 2024-08-31T22:28:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:31.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=163, tag_tid=23, entry_tid=9851' 2024-08-31T22:28:31.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:31.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9851 2024-08-31T22:28:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 9851 2024-08-31T22:28:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:31.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=9966], [object_number=161, tag_tid=23, entry_tid=9965], [object_number=160, tag_tid=23, entry_tid=9964], [object_number=163, tag_tid=23, entry_tid=9963]]], state=connected] 2024-08-31T22:28:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=9966], [object_number=161, tag_tid=23, entry_tid=9965], [object_number=160, tag_tid=23, entry_tid=9964], [object_number=163, tag_tid=23, entry_tid=9963]]], state=connected] 2024-08-31T22:28:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=162, tag_tid=23, entry_tid=9966' 2024-08-31T22:28:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=162, tag_tid=23, entry_tid=9966' 2024-08-31T22:28:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=9966' '!=' 'object_number=163, tag_tid=23, entry_tid=9851' 2024-08-31T22:28:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=9966' = 'object_number=163, tag_tid=23, entry_tid=9851' 2024-08-31T22:28:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=162, tag_tid=23, entry_tid=9966' 2024-08-31T22:28:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:31.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:31.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:31.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=9966' 2024-08-31T22:28:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=9966' 2024-08-31T22:28:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9966 2024-08-31T22:28:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 9966 2024-08-31T22:28:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=10049], [object_number=160, tag_tid=23, entry_tid=10048], [object_number=163, tag_tid=23, entry_tid=10047], [object_number=162, tag_tid=23, entry_tid=10046]]], state=connected] 2024-08-31T22:28:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=10049], [object_number=160, tag_tid=23, entry_tid=10048], [object_number=163, tag_tid=23, entry_tid=10047], [object_number=162, tag_tid=23, entry_tid=10046]]], state=connected] 2024-08-31T22:28:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=161, tag_tid=23, entry_tid=10049' 2024-08-31T22:28:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=161, tag_tid=23, entry_tid=10049' 2024-08-31T22:28:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=10049' '!=' 'object_number=162, tag_tid=23, entry_tid=9966' 2024-08-31T22:28:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=10049' = 'object_number=162, tag_tid=23, entry_tid=9966' 2024-08-31T22:28:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=161, tag_tid=23, entry_tid=10049' 2024-08-31T22:28:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=10049' 2024-08-31T22:28:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=10049' 2024-08-31T22:28:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10049 2024-08-31T22:28:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 10049 2024-08-31T22:28:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:32.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:32.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=10138], [object_number=161, tag_tid=23, entry_tid=10137], [object_number=160, tag_tid=23, entry_tid=10136], [object_number=163, tag_tid=23, entry_tid=10135]]], state=connected] 2024-08-31T22:28:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:32.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:32.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:32.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=10138], [object_number=161, tag_tid=23, entry_tid=10137], [object_number=160, tag_tid=23, entry_tid=10136], [object_number=163, tag_tid=23, entry_tid=10135]]], state=connected] 2024-08-31T22:28:32.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=162, tag_tid=23, entry_tid=10138' 2024-08-31T22:28:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=162, tag_tid=23, entry_tid=10138' 2024-08-31T22:28:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=10138' '!=' 'object_number=161, tag_tid=23, entry_tid=10049' 2024-08-31T22:28:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=10138' = 'object_number=161, tag_tid=23, entry_tid=10049' 2024-08-31T22:28:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=162, tag_tid=23, entry_tid=10138' 2024-08-31T22:28:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=10138' 2024-08-31T22:28:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:32.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=10138' 2024-08-31T22:28:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10138 2024-08-31T22:28:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 10138 2024-08-31T22:28:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=10210], [object_number=161, tag_tid=23, entry_tid=10209], [object_number=160, tag_tid=23, entry_tid=10208], [object_number=163, tag_tid=23, entry_tid=10207]]], state=connected] 2024-08-31T22:28:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:33.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:33.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:33.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:33.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=10210], [object_number=161, tag_tid=23, entry_tid=10209], [object_number=160, tag_tid=23, entry_tid=10208], [object_number=163, tag_tid=23, entry_tid=10207]]], state=connected] 2024-08-31T22:28:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=162, tag_tid=23, entry_tid=10210' 2024-08-31T22:28:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=162, tag_tid=23, entry_tid=10210' 2024-08-31T22:28:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=10210' '!=' 'object_number=162, tag_tid=23, entry_tid=10138' 2024-08-31T22:28:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=10210' = 'object_number=162, tag_tid=23, entry_tid=10138' 2024-08-31T22:28:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=162, tag_tid=23, entry_tid=10210' 2024-08-31T22:28:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=10210' 2024-08-31T22:28:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=10210' 2024-08-31T22:28:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10210 2024-08-31T22:28:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 10210 2024-08-31T22:28:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:33.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:33.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:33.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:33.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:33.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:33.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:33.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:33.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:33.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=160, tag_tid=23, entry_tid=10268], [object_number=163, tag_tid=23, entry_tid=10267], [object_number=162, tag_tid=23, entry_tid=10266], [object_number=161, tag_tid=23, entry_tid=10265]]], state=connected] 2024-08-31T22:28:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=160, tag_tid=23, entry_tid=10268], [object_number=163, tag_tid=23, entry_tid=10267], [object_number=162, tag_tid=23, entry_tid=10266], [object_number=161, tag_tid=23, entry_tid=10265]]], state=connected] 2024-08-31T22:28:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=160, tag_tid=23, entry_tid=10268' 2024-08-31T22:28:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=160, tag_tid=23, entry_tid=10268' 2024-08-31T22:28:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=160, tag_tid=23, entry_tid=10268' '!=' 'object_number=162, tag_tid=23, entry_tid=10210' 2024-08-31T22:28:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=160, tag_tid=23, entry_tid=10268' = 'object_number=162, tag_tid=23, entry_tid=10210' 2024-08-31T22:28:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=160, tag_tid=23, entry_tid=10268' 2024-08-31T22:28:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=160, tag_tid=23, entry_tid=10268' 2024-08-31T22:28:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=160, tag_tid=23, entry_tid=10268' 2024-08-31T22:28:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10268 2024-08-31T22:28:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 10268 2024-08-31T22:28:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:34.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:34.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=10349], [object_number=160, tag_tid=23, entry_tid=10348], [object_number=163, tag_tid=23, entry_tid=10347], [object_number=162, tag_tid=23, entry_tid=10346]]], state=connected] 2024-08-31T22:28:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=10349], [object_number=160, tag_tid=23, entry_tid=10348], [object_number=163, tag_tid=23, entry_tid=10347], [object_number=162, tag_tid=23, entry_tid=10346]]], state=connected] 2024-08-31T22:28:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=161, tag_tid=23, entry_tid=10349' 2024-08-31T22:28:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=161, tag_tid=23, entry_tid=10349' 2024-08-31T22:28:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=10349' '!=' 'object_number=160, tag_tid=23, entry_tid=10268' 2024-08-31T22:28:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=10349' = 'object_number=160, tag_tid=23, entry_tid=10268' 2024-08-31T22:28:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=161, tag_tid=23, entry_tid=10349' 2024-08-31T22:28:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=10349' 2024-08-31T22:28:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=10349' 2024-08-31T22:28:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10349 2024-08-31T22:28:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 10349 2024-08-31T22:28:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=160, tag_tid=23, entry_tid=10468], [object_number=163, tag_tid=23, entry_tid=10467], [object_number=162, tag_tid=23, entry_tid=10466], [object_number=161, tag_tid=23, entry_tid=10465]]], state=connected] 2024-08-31T22:28:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=160, tag_tid=23, entry_tid=10468], [object_number=163, tag_tid=23, entry_tid=10467], [object_number=162, tag_tid=23, entry_tid=10466], [object_number=161, tag_tid=23, entry_tid=10465]]], state=connected] 2024-08-31T22:28:34.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=160, tag_tid=23, entry_tid=10468' 2024-08-31T22:28:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=160, tag_tid=23, entry_tid=10468' 2024-08-31T22:28:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=160, tag_tid=23, entry_tid=10468' '!=' 'object_number=161, tag_tid=23, entry_tid=10349' 2024-08-31T22:28:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=160, tag_tid=23, entry_tid=10468' = 'object_number=161, tag_tid=23, entry_tid=10349' 2024-08-31T22:28:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=160, tag_tid=23, entry_tid=10468' 2024-08-31T22:28:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=160, tag_tid=23, entry_tid=10468' 2024-08-31T22:28:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:34.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:34.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:34.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:34.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:34.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=160, tag_tid=23, entry_tid=10468' 2024-08-31T22:28:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10468 2024-08-31T22:28:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 10468 2024-08-31T22:28:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=163, tag_tid=23, entry_tid=10559], [object_number=162, tag_tid=23, entry_tid=10558], [object_number=161, tag_tid=23, entry_tid=10557], [object_number=160, tag_tid=23, entry_tid=10556]]], state=connected] 2024-08-31T22:28:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=163, tag_tid=23, entry_tid=10559], [object_number=162, tag_tid=23, entry_tid=10558], [object_number=161, tag_tid=23, entry_tid=10557], [object_number=160, tag_tid=23, entry_tid=10556]]], state=connected] 2024-08-31T22:28:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=163, tag_tid=23, entry_tid=10559' 2024-08-31T22:28:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=163, tag_tid=23, entry_tid=10559' 2024-08-31T22:28:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=163, tag_tid=23, entry_tid=10559' '!=' 'object_number=160, tag_tid=23, entry_tid=10468' 2024-08-31T22:28:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=163, tag_tid=23, entry_tid=10559' = 'object_number=160, tag_tid=23, entry_tid=10468' 2024-08-31T22:28:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=163, tag_tid=23, entry_tid=10559' 2024-08-31T22:28:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=163, tag_tid=23, entry_tid=10559' 2024-08-31T22:28:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=163, tag_tid=23, entry_tid=10559' 2024-08-31T22:28:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10559 2024-08-31T22:28:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 10559 2024-08-31T22:28:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=163, tag_tid=23, entry_tid=10655], [object_number=162, tag_tid=23, entry_tid=10654], [object_number=161, tag_tid=23, entry_tid=10653], [object_number=160, tag_tid=23, entry_tid=10652]]], state=connected] 2024-08-31T22:28:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=163, tag_tid=23, entry_tid=10655], [object_number=162, tag_tid=23, entry_tid=10654], [object_number=161, tag_tid=23, entry_tid=10653], [object_number=160, tag_tid=23, entry_tid=10652]]], state=connected] 2024-08-31T22:28:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=163, tag_tid=23, entry_tid=10655' 2024-08-31T22:28:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=163, tag_tid=23, entry_tid=10655' 2024-08-31T22:28:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=163, tag_tid=23, entry_tid=10655' '!=' 'object_number=163, tag_tid=23, entry_tid=10559' 2024-08-31T22:28:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=163, tag_tid=23, entry_tid=10655' = 'object_number=163, tag_tid=23, entry_tid=10559' 2024-08-31T22:28:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=163, tag_tid=23, entry_tid=10655' 2024-08-31T22:28:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:35.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=163, tag_tid=23, entry_tid=10655' 2024-08-31T22:28:35.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:35.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=163, tag_tid=23, entry_tid=10655' 2024-08-31T22:28:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10655 2024-08-31T22:28:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 10655 2024-08-31T22:28:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:36.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:36.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:36.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:36.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=10738], [object_number=161, tag_tid=23, entry_tid=10737], [object_number=160, tag_tid=23, entry_tid=10736], [object_number=163, tag_tid=23, entry_tid=10735]]], state=connected] 2024-08-31T22:28:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:36.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:36.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:36.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=10738], [object_number=161, tag_tid=23, entry_tid=10737], [object_number=160, tag_tid=23, entry_tid=10736], [object_number=163, tag_tid=23, entry_tid=10735]]], state=connected] 2024-08-31T22:28:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=162, tag_tid=23, entry_tid=10738' 2024-08-31T22:28:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=162, tag_tid=23, entry_tid=10738' 2024-08-31T22:28:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=10738' '!=' 'object_number=163, tag_tid=23, entry_tid=10655' 2024-08-31T22:28:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=10738' = 'object_number=163, tag_tid=23, entry_tid=10655' 2024-08-31T22:28:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=162, tag_tid=23, entry_tid=10738' 2024-08-31T22:28:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=10738' 2024-08-31T22:28:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=10738' 2024-08-31T22:28:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10738 2024-08-31T22:28:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 10738 2024-08-31T22:28:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=10818], [object_number=161, tag_tid=23, entry_tid=10817], [object_number=160, tag_tid=23, entry_tid=10816], [object_number=163, tag_tid=23, entry_tid=10815]]], state=connected] 2024-08-31T22:28:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=10818], [object_number=161, tag_tid=23, entry_tid=10817], [object_number=160, tag_tid=23, entry_tid=10816], [object_number=163, tag_tid=23, entry_tid=10815]]], state=connected] 2024-08-31T22:28:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=162, tag_tid=23, entry_tid=10818' 2024-08-31T22:28:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=162, tag_tid=23, entry_tid=10818' 2024-08-31T22:28:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=10818' '!=' 'object_number=162, tag_tid=23, entry_tid=10738' 2024-08-31T22:28:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=10818' = 'object_number=162, tag_tid=23, entry_tid=10738' 2024-08-31T22:28:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=162, tag_tid=23, entry_tid=10818' 2024-08-31T22:28:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=10818' 2024-08-31T22:28:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=10818' 2024-08-31T22:28:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10818 2024-08-31T22:28:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 10818 2024-08-31T22:28:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=160, tag_tid=23, entry_tid=10900], [object_number=163, tag_tid=23, entry_tid=10899], [object_number=162, tag_tid=23, entry_tid=10898], [object_number=161, tag_tid=23, entry_tid=10897]]], state=connected] 2024-08-31T22:28:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=160, tag_tid=23, entry_tid=10900], [object_number=163, tag_tid=23, entry_tid=10899], [object_number=162, tag_tid=23, entry_tid=10898], [object_number=161, tag_tid=23, entry_tid=10897]]], state=connected] 2024-08-31T22:28:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=160, tag_tid=23, entry_tid=10900' 2024-08-31T22:28:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=160, tag_tid=23, entry_tid=10900' 2024-08-31T22:28:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=160, tag_tid=23, entry_tid=10900' '!=' 'object_number=162, tag_tid=23, entry_tid=10818' 2024-08-31T22:28:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=160, tag_tid=23, entry_tid=10900' = 'object_number=162, tag_tid=23, entry_tid=10818' 2024-08-31T22:28:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=160, tag_tid=23, entry_tid=10900' 2024-08-31T22:28:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=160, tag_tid=23, entry_tid=10900' 2024-08-31T22:28:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=160, tag_tid=23, entry_tid=10900' 2024-08-31T22:28:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10900 2024-08-31T22:28:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 10900 2024-08-31T22:28:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=10978], [object_number=161, tag_tid=23, entry_tid=10977], [object_number=160, tag_tid=23, entry_tid=10976], [object_number=163, tag_tid=23, entry_tid=10975]]], state=connected] 2024-08-31T22:28:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:37.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:37.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=10978], [object_number=161, tag_tid=23, entry_tid=10977], [object_number=160, tag_tid=23, entry_tid=10976], [object_number=163, tag_tid=23, entry_tid=10975]]], state=connected] 2024-08-31T22:28:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=162, tag_tid=23, entry_tid=10978' 2024-08-31T22:28:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=162, tag_tid=23, entry_tid=10978' 2024-08-31T22:28:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=10978' '!=' 'object_number=160, tag_tid=23, entry_tid=10900' 2024-08-31T22:28:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=10978' = 'object_number=160, tag_tid=23, entry_tid=10900' 2024-08-31T22:28:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=162, tag_tid=23, entry_tid=10978' 2024-08-31T22:28:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=10978' 2024-08-31T22:28:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:37.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=10978' 2024-08-31T22:28:37.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:37.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:37.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=10978 2024-08-31T22:28:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 10978 2024-08-31T22:28:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:38.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:38.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=163, tag_tid=23, entry_tid=11067], [object_number=162, tag_tid=23, entry_tid=11066], [object_number=161, tag_tid=23, entry_tid=11065], [object_number=160, tag_tid=23, entry_tid=11064]]], state=connected] 2024-08-31T22:28:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=163, tag_tid=23, entry_tid=11067], [object_number=162, tag_tid=23, entry_tid=11066], [object_number=161, tag_tid=23, entry_tid=11065], [object_number=160, tag_tid=23, entry_tid=11064]]], state=connected] 2024-08-31T22:28:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=163, tag_tid=23, entry_tid=11067' 2024-08-31T22:28:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=163, tag_tid=23, entry_tid=11067' 2024-08-31T22:28:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=163, tag_tid=23, entry_tid=11067' '!=' 'object_number=162, tag_tid=23, entry_tid=10978' 2024-08-31T22:28:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=163, tag_tid=23, entry_tid=11067' = 'object_number=162, tag_tid=23, entry_tid=10978' 2024-08-31T22:28:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=163, tag_tid=23, entry_tid=11067' 2024-08-31T22:28:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=163, tag_tid=23, entry_tid=11067' 2024-08-31T22:28:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=163, tag_tid=23, entry_tid=11067' 2024-08-31T22:28:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11067 2024-08-31T22:28:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 11067 2024-08-31T22:28:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:38.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:38.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:38.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=163, tag_tid=23, entry_tid=11147], [object_number=162, tag_tid=23, entry_tid=11146], [object_number=161, tag_tid=23, entry_tid=11145], [object_number=160, tag_tid=23, entry_tid=11144]]], state=connected] 2024-08-31T22:28:38.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=163, tag_tid=23, entry_tid=11147], [object_number=162, tag_tid=23, entry_tid=11146], [object_number=161, tag_tid=23, entry_tid=11145], [object_number=160, tag_tid=23, entry_tid=11144]]], state=connected] 2024-08-31T22:28:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=163, tag_tid=23, entry_tid=11147' 2024-08-31T22:28:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=163, tag_tid=23, entry_tid=11147' 2024-08-31T22:28:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=163, tag_tid=23, entry_tid=11147' '!=' 'object_number=163, tag_tid=23, entry_tid=11067' 2024-08-31T22:28:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=163, tag_tid=23, entry_tid=11147' = 'object_number=163, tag_tid=23, entry_tid=11067' 2024-08-31T22:28:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=163, tag_tid=23, entry_tid=11147' 2024-08-31T22:28:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=163, tag_tid=23, entry_tid=11147' 2024-08-31T22:28:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:38.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=163, tag_tid=23, entry_tid=11147' 2024-08-31T22:28:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:38.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11147 2024-08-31T22:28:38.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 11147 2024-08-31T22:28:38.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:38.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:38.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:39.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:39.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=160, tag_tid=23, entry_tid=11228], [object_number=163, tag_tid=23, entry_tid=11227], [object_number=162, tag_tid=23, entry_tid=11226], [object_number=161, tag_tid=23, entry_tid=11225]]], state=connected] 2024-08-31T22:28:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=160, tag_tid=23, entry_tid=11228], [object_number=163, tag_tid=23, entry_tid=11227], [object_number=162, tag_tid=23, entry_tid=11226], [object_number=161, tag_tid=23, entry_tid=11225]]], state=connected] 2024-08-31T22:28:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=160, tag_tid=23, entry_tid=11228' 2024-08-31T22:28:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=160, tag_tid=23, entry_tid=11228' 2024-08-31T22:28:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=160, tag_tid=23, entry_tid=11228' '!=' 'object_number=163, tag_tid=23, entry_tid=11147' 2024-08-31T22:28:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=160, tag_tid=23, entry_tid=11228' = 'object_number=163, tag_tid=23, entry_tid=11147' 2024-08-31T22:28:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=160, tag_tid=23, entry_tid=11228' 2024-08-31T22:28:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=160, tag_tid=23, entry_tid=11228' 2024-08-31T22:28:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=160, tag_tid=23, entry_tid=11228' 2024-08-31T22:28:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11228 2024-08-31T22:28:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 11228 2024-08-31T22:28:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:39.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:39.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:39.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:39.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:39.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=11313], [object_number=160, tag_tid=23, entry_tid=11312], [object_number=163, tag_tid=23, entry_tid=11311], [object_number=162, tag_tid=23, entry_tid=11310]]], state=connected] 2024-08-31T22:28:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:40.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=11313], [object_number=160, tag_tid=23, entry_tid=11312], [object_number=163, tag_tid=23, entry_tid=11311], [object_number=162, tag_tid=23, entry_tid=11310]]], state=connected] 2024-08-31T22:28:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=161, tag_tid=23, entry_tid=11313' 2024-08-31T22:28:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=161, tag_tid=23, entry_tid=11313' 2024-08-31T22:28:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=11313' '!=' 'object_number=160, tag_tid=23, entry_tid=11228' 2024-08-31T22:28:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=11313' = 'object_number=160, tag_tid=23, entry_tid=11228' 2024-08-31T22:28:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=161, tag_tid=23, entry_tid=11313' 2024-08-31T22:28:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=11313' 2024-08-31T22:28:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=11313' 2024-08-31T22:28:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11313 2024-08-31T22:28:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 11313 2024-08-31T22:28:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=11434], [object_number=161, tag_tid=23, entry_tid=11433], [object_number=160, tag_tid=23, entry_tid=11432], [object_number=163, tag_tid=23, entry_tid=11431]]], state=connected] 2024-08-31T22:28:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=11434], [object_number=161, tag_tid=23, entry_tid=11433], [object_number=160, tag_tid=23, entry_tid=11432], [object_number=163, tag_tid=23, entry_tid=11431]]], state=connected] 2024-08-31T22:28:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=162, tag_tid=23, entry_tid=11434' 2024-08-31T22:28:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=162, tag_tid=23, entry_tid=11434' 2024-08-31T22:28:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=11434' '!=' 'object_number=161, tag_tid=23, entry_tid=11313' 2024-08-31T22:28:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=11434' = 'object_number=161, tag_tid=23, entry_tid=11313' 2024-08-31T22:28:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=162, tag_tid=23, entry_tid=11434' 2024-08-31T22:28:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=11434' 2024-08-31T22:28:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=11434' 2024-08-31T22:28:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11434 2024-08-31T22:28:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 11434 2024-08-31T22:28:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=11510], [object_number=161, tag_tid=23, entry_tid=11509], [object_number=160, tag_tid=23, entry_tid=11508], [object_number=163, tag_tid=23, entry_tid=11507]]], state=connected] 2024-08-31T22:28:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=11510], [object_number=161, tag_tid=23, entry_tid=11509], [object_number=160, tag_tid=23, entry_tid=11508], [object_number=163, tag_tid=23, entry_tid=11507]]], state=connected] 2024-08-31T22:28:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=162, tag_tid=23, entry_tid=11510' 2024-08-31T22:28:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=162, tag_tid=23, entry_tid=11510' 2024-08-31T22:28:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=11510' '!=' 'object_number=162, tag_tid=23, entry_tid=11434' 2024-08-31T22:28:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=11510' = 'object_number=162, tag_tid=23, entry_tid=11434' 2024-08-31T22:28:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=162, tag_tid=23, entry_tid=11510' 2024-08-31T22:28:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=11510' 2024-08-31T22:28:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=11510' 2024-08-31T22:28:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11510 2024-08-31T22:28:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 11510 2024-08-31T22:28:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=11585], [object_number=160, tag_tid=23, entry_tid=11584], [object_number=163, tag_tid=23, entry_tid=11583], [object_number=162, tag_tid=23, entry_tid=11582]]], state=connected] 2024-08-31T22:28:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=11585], [object_number=160, tag_tid=23, entry_tid=11584], [object_number=163, tag_tid=23, entry_tid=11583], [object_number=162, tag_tid=23, entry_tid=11582]]], state=connected] 2024-08-31T22:28:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=161, tag_tid=23, entry_tid=11585' 2024-08-31T22:28:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=161, tag_tid=23, entry_tid=11585' 2024-08-31T22:28:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=11585' '!=' 'object_number=162, tag_tid=23, entry_tid=11510' 2024-08-31T22:28:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=11585' = 'object_number=162, tag_tid=23, entry_tid=11510' 2024-08-31T22:28:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=161, tag_tid=23, entry_tid=11585' 2024-08-31T22:28:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=11585' 2024-08-31T22:28:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=11585' 2024-08-31T22:28:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11585 2024-08-31T22:28:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 11585 2024-08-31T22:28:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:42.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:42.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:42.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:42.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:42.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=11710], [object_number=161, tag_tid=23, entry_tid=11709], [object_number=160, tag_tid=23, entry_tid=11708], [object_number=163, tag_tid=23, entry_tid=11707]]], state=connected] 2024-08-31T22:28:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=162, tag_tid=23, entry_tid=11710], [object_number=161, tag_tid=23, entry_tid=11709], [object_number=160, tag_tid=23, entry_tid=11708], [object_number=163, tag_tid=23, entry_tid=11707]]], state=connected] 2024-08-31T22:28:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=162, tag_tid=23, entry_tid=11710' 2024-08-31T22:28:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=162, tag_tid=23, entry_tid=11710' 2024-08-31T22:28:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=11710' '!=' 'object_number=161, tag_tid=23, entry_tid=11585' 2024-08-31T22:28:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=162, tag_tid=23, entry_tid=11710' = 'object_number=161, tag_tid=23, entry_tid=11585' 2024-08-31T22:28:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=162, tag_tid=23, entry_tid=11710' 2024-08-31T22:28:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=11710' 2024-08-31T22:28:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=162, tag_tid=23, entry_tid=11710' 2024-08-31T22:28:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11710 2024-08-31T22:28:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 11710 2024-08-31T22:28:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:42.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:42.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=11809], [object_number=160, tag_tid=23, entry_tid=11808], [object_number=163, tag_tid=23, entry_tid=11807], [object_number=162, tag_tid=23, entry_tid=11806]]], state=connected] 2024-08-31T22:28:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=11809], [object_number=160, tag_tid=23, entry_tid=11808], [object_number=163, tag_tid=23, entry_tid=11807], [object_number=162, tag_tid=23, entry_tid=11806]]], state=connected] 2024-08-31T22:28:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=161, tag_tid=23, entry_tid=11809' 2024-08-31T22:28:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=161, tag_tid=23, entry_tid=11809' 2024-08-31T22:28:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=11809' '!=' 'object_number=162, tag_tid=23, entry_tid=11710' 2024-08-31T22:28:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=11809' = 'object_number=162, tag_tid=23, entry_tid=11710' 2024-08-31T22:28:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=161, tag_tid=23, entry_tid=11809' 2024-08-31T22:28:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=11809' 2024-08-31T22:28:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=11809' 2024-08-31T22:28:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11809 2024-08-31T22:28:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 11809 2024-08-31T22:28:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:43.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:43.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:43.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:43.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=160, tag_tid=23, entry_tid=11944], [object_number=163, tag_tid=23, entry_tid=11943], [object_number=162, tag_tid=23, entry_tid=11942], [object_number=161, tag_tid=23, entry_tid=11941]]], state=connected] 2024-08-31T22:28:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=160, tag_tid=23, entry_tid=11944], [object_number=163, tag_tid=23, entry_tid=11943], [object_number=162, tag_tid=23, entry_tid=11942], [object_number=161, tag_tid=23, entry_tid=11941]]], state=connected] 2024-08-31T22:28:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=160, tag_tid=23, entry_tid=11944' 2024-08-31T22:28:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=160, tag_tid=23, entry_tid=11944' 2024-08-31T22:28:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=160, tag_tid=23, entry_tid=11944' '!=' 'object_number=161, tag_tid=23, entry_tid=11809' 2024-08-31T22:28:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=160, tag_tid=23, entry_tid=11944' = 'object_number=161, tag_tid=23, entry_tid=11809' 2024-08-31T22:28:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=160, tag_tid=23, entry_tid=11944' 2024-08-31T22:28:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=160, tag_tid=23, entry_tid=11944' 2024-08-31T22:28:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=160, tag_tid=23, entry_tid=11944' 2024-08-31T22:28:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11944 2024-08-31T22:28:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 11944 2024-08-31T22:28:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:43.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:43.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:43.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:43.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:43.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=12017], [object_number=164, tag_tid=23, entry_tid=12016], [object_number=167, tag_tid=23, entry_tid=12015], [object_number=162, tag_tid=23, entry_tid=12014]]], state=connected] 2024-08-31T22:28:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=161, tag_tid=23, entry_tid=12017], [object_number=164, tag_tid=23, entry_tid=12016], [object_number=167, tag_tid=23, entry_tid=12015], [object_number=162, tag_tid=23, entry_tid=12014]]], state=connected] 2024-08-31T22:28:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=161, tag_tid=23, entry_tid=12017' 2024-08-31T22:28:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=161, tag_tid=23, entry_tid=12017' 2024-08-31T22:28:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=12017' '!=' 'object_number=160, tag_tid=23, entry_tid=11944' 2024-08-31T22:28:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=161, tag_tid=23, entry_tid=12017' = 'object_number=160, tag_tid=23, entry_tid=11944' 2024-08-31T22:28:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=161, tag_tid=23, entry_tid=12017' 2024-08-31T22:28:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=12017' 2024-08-31T22:28:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=161, tag_tid=23, entry_tid=12017' 2024-08-31T22:28:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12017 2024-08-31T22:28:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 12017 2024-08-31T22:28:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=165, tag_tid=23, entry_tid=12097], [object_number=164, tag_tid=23, entry_tid=12096], [object_number=167, tag_tid=23, entry_tid=12095], [object_number=162, tag_tid=23, entry_tid=12094]]], state=connected] 2024-08-31T22:28:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 40 2024-08-31T22:28:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=165, tag_tid=23, entry_tid=12097], [object_number=164, tag_tid=23, entry_tid=12096], [object_number=167, tag_tid=23, entry_tid=12095], [object_number=162, tag_tid=23, entry_tid=12094]]], state=connected] 2024-08-31T22:28:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=165, tag_tid=23, entry_tid=12097' 2024-08-31T22:28:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=165, tag_tid=23, entry_tid=12097' 2024-08-31T22:28:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=165, tag_tid=23, entry_tid=12097' '!=' 'object_number=161, tag_tid=23, entry_tid=12017' 2024-08-31T22:28:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=165, tag_tid=23, entry_tid=12097' = 'object_number=161, tag_tid=23, entry_tid=12017' 2024-08-31T22:28:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=165, tag_tid=23, entry_tid=12097' 2024-08-31T22:28:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:44.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=23, entry_tid=12097' 2024-08-31T22:28:44.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:44.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=23, entry_tid=12097' 2024-08-31T22:28:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12097 2024-08-31T22:28:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 12097 2024-08-31T22:28:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=167, tag_tid=23, entry_tid=12179], [object_number=166, tag_tid=23, entry_tid=12178], [object_number=165, tag_tid=23, entry_tid=12177], [object_number=164, tag_tid=23, entry_tid=12176]]], state=connected] 2024-08-31T22:28:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:44.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:44.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:44.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:44.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:44.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=167, tag_tid=23, entry_tid=12179], [object_number=166, tag_tid=23, entry_tid=12178], [object_number=165, tag_tid=23, entry_tid=12177], [object_number=164, tag_tid=23, entry_tid=12176]]], state=connected] 2024-08-31T22:28:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=167, tag_tid=23, entry_tid=12179' 2024-08-31T22:28:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=167, tag_tid=23, entry_tid=12179' 2024-08-31T22:28:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=167, tag_tid=23, entry_tid=12179' '!=' 'object_number=165, tag_tid=23, entry_tid=12097' 2024-08-31T22:28:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=167, tag_tid=23, entry_tid=12179' = 'object_number=165, tag_tid=23, entry_tid=12097' 2024-08-31T22:28:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=167, tag_tid=23, entry_tid=12179' 2024-08-31T22:28:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=167, tag_tid=23, entry_tid=12179' 2024-08-31T22:28:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=167, tag_tid=23, entry_tid=12179' 2024-08-31T22:28:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12179 2024-08-31T22:28:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 12179 2024-08-31T22:28:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=164, tag_tid=23, entry_tid=12284], [object_number=167, tag_tid=23, entry_tid=12283], [object_number=166, tag_tid=23, entry_tid=12282], [object_number=165, tag_tid=23, entry_tid=12281]]], state=connected] 2024-08-31T22:28:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=164, tag_tid=23, entry_tid=12284], [object_number=167, tag_tid=23, entry_tid=12283], [object_number=166, tag_tid=23, entry_tid=12282], [object_number=165, tag_tid=23, entry_tid=12281]]], state=connected] 2024-08-31T22:28:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=164, tag_tid=23, entry_tid=12284' 2024-08-31T22:28:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=164, tag_tid=23, entry_tid=12284' 2024-08-31T22:28:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=164, tag_tid=23, entry_tid=12284' '!=' 'object_number=167, tag_tid=23, entry_tid=12179' 2024-08-31T22:28:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=164, tag_tid=23, entry_tid=12284' = 'object_number=167, tag_tid=23, entry_tid=12179' 2024-08-31T22:28:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=164, tag_tid=23, entry_tid=12284' 2024-08-31T22:28:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=164, tag_tid=23, entry_tid=12284' 2024-08-31T22:28:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:45.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=164, tag_tid=23, entry_tid=12284' 2024-08-31T22:28:45.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:45.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12284 2024-08-31T22:28:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 12284 2024-08-31T22:28:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:45.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:45.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=165, tag_tid=23, entry_tid=12389], [object_number=164, tag_tid=23, entry_tid=12388], [object_number=167, tag_tid=23, entry_tid=12387], [object_number=166, tag_tid=23, entry_tid=12386]]], state=connected] 2024-08-31T22:28:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:46.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:46.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:46.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:46.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:46.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=165, tag_tid=23, entry_tid=12389], [object_number=164, tag_tid=23, entry_tid=12388], [object_number=167, tag_tid=23, entry_tid=12387], [object_number=166, tag_tid=23, entry_tid=12386]]], state=connected] 2024-08-31T22:28:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=165, tag_tid=23, entry_tid=12389' 2024-08-31T22:28:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=165, tag_tid=23, entry_tid=12389' 2024-08-31T22:28:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=165, tag_tid=23, entry_tid=12389' '!=' 'object_number=164, tag_tid=23, entry_tid=12284' 2024-08-31T22:28:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=165, tag_tid=23, entry_tid=12389' = 'object_number=164, tag_tid=23, entry_tid=12284' 2024-08-31T22:28:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=165, tag_tid=23, entry_tid=12389' 2024-08-31T22:28:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=23, entry_tid=12389' 2024-08-31T22:28:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=23, entry_tid=12389' 2024-08-31T22:28:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12389 2024-08-31T22:28:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 12389 2024-08-31T22:28:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:46.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:46.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:46.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:46.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:46.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:46.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=164, tag_tid=23, entry_tid=12512], [object_number=167, tag_tid=23, entry_tid=12511], [object_number=166, tag_tid=23, entry_tid=12510], [object_number=165, tag_tid=23, entry_tid=12509]]], state=connected] 2024-08-31T22:28:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:46.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=164, tag_tid=23, entry_tid=12512], [object_number=167, tag_tid=23, entry_tid=12511], [object_number=166, tag_tid=23, entry_tid=12510], [object_number=165, tag_tid=23, entry_tid=12509]]], state=connected] 2024-08-31T22:28:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=164, tag_tid=23, entry_tid=12512' 2024-08-31T22:28:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=164, tag_tid=23, entry_tid=12512' 2024-08-31T22:28:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=164, tag_tid=23, entry_tid=12512' '!=' 'object_number=165, tag_tid=23, entry_tid=12389' 2024-08-31T22:28:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=164, tag_tid=23, entry_tid=12512' = 'object_number=165, tag_tid=23, entry_tid=12389' 2024-08-31T22:28:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=164, tag_tid=23, entry_tid=12512' 2024-08-31T22:28:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=164, tag_tid=23, entry_tid=12512' 2024-08-31T22:28:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=164, tag_tid=23, entry_tid=12512' 2024-08-31T22:28:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12512 2024-08-31T22:28:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 12512 2024-08-31T22:28:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=164, tag_tid=23, entry_tid=12592], [object_number=167, tag_tid=23, entry_tid=12591], [object_number=166, tag_tid=23, entry_tid=12590], [object_number=165, tag_tid=23, entry_tid=12589]]], state=connected] 2024-08-31T22:28:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=164, tag_tid=23, entry_tid=12592], [object_number=167, tag_tid=23, entry_tid=12591], [object_number=166, tag_tid=23, entry_tid=12590], [object_number=165, tag_tid=23, entry_tid=12589]]], state=connected] 2024-08-31T22:28:47.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=164, tag_tid=23, entry_tid=12592' 2024-08-31T22:28:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=164, tag_tid=23, entry_tid=12592' 2024-08-31T22:28:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=164, tag_tid=23, entry_tid=12592' '!=' 'object_number=164, tag_tid=23, entry_tid=12512' 2024-08-31T22:28:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=164, tag_tid=23, entry_tid=12592' = 'object_number=164, tag_tid=23, entry_tid=12512' 2024-08-31T22:28:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=164, tag_tid=23, entry_tid=12592' 2024-08-31T22:28:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=164, tag_tid=23, entry_tid=12592' 2024-08-31T22:28:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=164, tag_tid=23, entry_tid=12592' 2024-08-31T22:28:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12592 2024-08-31T22:28:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 12592 2024-08-31T22:28:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=164, tag_tid=23, entry_tid=12676], [object_number=167, tag_tid=23, entry_tid=12675], [object_number=166, tag_tid=23, entry_tid=12674], [object_number=165, tag_tid=23, entry_tid=12673]]], state=connected] 2024-08-31T22:28:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=164, tag_tid=23, entry_tid=12676], [object_number=167, tag_tid=23, entry_tid=12675], [object_number=166, tag_tid=23, entry_tid=12674], [object_number=165, tag_tid=23, entry_tid=12673]]], state=connected] 2024-08-31T22:28:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=164, tag_tid=23, entry_tid=12676' 2024-08-31T22:28:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=164, tag_tid=23, entry_tid=12676' 2024-08-31T22:28:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=164, tag_tid=23, entry_tid=12676' '!=' 'object_number=164, tag_tid=23, entry_tid=12592' 2024-08-31T22:28:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=164, tag_tid=23, entry_tid=12676' = 'object_number=164, tag_tid=23, entry_tid=12592' 2024-08-31T22:28:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=164, tag_tid=23, entry_tid=12676' 2024-08-31T22:28:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=164, tag_tid=23, entry_tid=12676' 2024-08-31T22:28:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=164, tag_tid=23, entry_tid=12676' 2024-08-31T22:28:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12676 2024-08-31T22:28:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 12676 2024-08-31T22:28:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=164, tag_tid=23, entry_tid=12760], [object_number=167, tag_tid=23, entry_tid=12759], [object_number=166, tag_tid=23, entry_tid=12758], [object_number=165, tag_tid=23, entry_tid=12757]]], state=connected] 2024-08-31T22:28:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:48.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:48.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:48.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:48.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:48.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:48.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:48.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:48.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=164, tag_tid=23, entry_tid=12760], [object_number=167, tag_tid=23, entry_tid=12759], [object_number=166, tag_tid=23, entry_tid=12758], [object_number=165, tag_tid=23, entry_tid=12757]]], state=connected] 2024-08-31T22:28:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=164, tag_tid=23, entry_tid=12760' 2024-08-31T22:28:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=164, tag_tid=23, entry_tid=12760' 2024-08-31T22:28:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=164, tag_tid=23, entry_tid=12760' '!=' 'object_number=164, tag_tid=23, entry_tid=12676' 2024-08-31T22:28:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=164, tag_tid=23, entry_tid=12760' = 'object_number=164, tag_tid=23, entry_tid=12676' 2024-08-31T22:28:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=164, tag_tid=23, entry_tid=12760' 2024-08-31T22:28:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=164, tag_tid=23, entry_tid=12760' 2024-08-31T22:28:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=164, tag_tid=23, entry_tid=12760' 2024-08-31T22:28:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12760 2024-08-31T22:28:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 12760 2024-08-31T22:28:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=165, tag_tid=23, entry_tid=12841], [object_number=164, tag_tid=23, entry_tid=12840], [object_number=167, tag_tid=23, entry_tid=12839], [object_number=166, tag_tid=23, entry_tid=12838]]], state=connected] 2024-08-31T22:28:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=165, tag_tid=23, entry_tid=12841], [object_number=164, tag_tid=23, entry_tid=12840], [object_number=167, tag_tid=23, entry_tid=12839], [object_number=166, tag_tid=23, entry_tid=12838]]], state=connected] 2024-08-31T22:28:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=165, tag_tid=23, entry_tid=12841' 2024-08-31T22:28:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=165, tag_tid=23, entry_tid=12841' 2024-08-31T22:28:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=165, tag_tid=23, entry_tid=12841' '!=' 'object_number=164, tag_tid=23, entry_tid=12760' 2024-08-31T22:28:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=165, tag_tid=23, entry_tid=12841' = 'object_number=164, tag_tid=23, entry_tid=12760' 2024-08-31T22:28:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=165, tag_tid=23, entry_tid=12841' 2024-08-31T22:28:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=23, entry_tid=12841' 2024-08-31T22:28:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=23, entry_tid=12841' 2024-08-31T22:28:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12841 2024-08-31T22:28:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 12841 2024-08-31T22:28:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:49.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=167, tag_tid=23, entry_tid=12923], [object_number=166, tag_tid=23, entry_tid=12922], [object_number=165, tag_tid=23, entry_tid=12921], [object_number=164, tag_tid=23, entry_tid=12920]]], state=connected] 2024-08-31T22:28:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=167, tag_tid=23, entry_tid=12923], [object_number=166, tag_tid=23, entry_tid=12922], [object_number=165, tag_tid=23, entry_tid=12921], [object_number=164, tag_tid=23, entry_tid=12920]]], state=connected] 2024-08-31T22:28:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=167, tag_tid=23, entry_tid=12923' 2024-08-31T22:28:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=167, tag_tid=23, entry_tid=12923' 2024-08-31T22:28:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=167, tag_tid=23, entry_tid=12923' '!=' 'object_number=165, tag_tid=23, entry_tid=12841' 2024-08-31T22:28:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=167, tag_tid=23, entry_tid=12923' = 'object_number=165, tag_tid=23, entry_tid=12841' 2024-08-31T22:28:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=167, tag_tid=23, entry_tid=12923' 2024-08-31T22:28:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=167, tag_tid=23, entry_tid=12923' 2024-08-31T22:28:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=167, tag_tid=23, entry_tid=12923' 2024-08-31T22:28:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12923 2024-08-31T22:28:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 12923 2024-08-31T22:28:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:49.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=164, tag_tid=23, entry_tid=13008], [object_number=167, tag_tid=23, entry_tid=13007], [object_number=166, tag_tid=23, entry_tid=13006], [object_number=165, tag_tid=23, entry_tid=13005]]], state=connected] 2024-08-31T22:28:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=164, tag_tid=23, entry_tid=13008], [object_number=167, tag_tid=23, entry_tid=13007], [object_number=166, tag_tid=23, entry_tid=13006], [object_number=165, tag_tid=23, entry_tid=13005]]], state=connected] 2024-08-31T22:28:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=164, tag_tid=23, entry_tid=13008' 2024-08-31T22:28:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=164, tag_tid=23, entry_tid=13008' 2024-08-31T22:28:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=164, tag_tid=23, entry_tid=13008' '!=' 'object_number=167, tag_tid=23, entry_tid=12923' 2024-08-31T22:28:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=164, tag_tid=23, entry_tid=13008' = 'object_number=167, tag_tid=23, entry_tid=12923' 2024-08-31T22:28:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=164, tag_tid=23, entry_tid=13008' 2024-08-31T22:28:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=164, tag_tid=23, entry_tid=13008' 2024-08-31T22:28:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=164, tag_tid=23, entry_tid=13008' 2024-08-31T22:28:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13008 2024-08-31T22:28:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 13008 2024-08-31T22:28:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=167, tag_tid=23, entry_tid=13087], [object_number=166, tag_tid=23, entry_tid=13086], [object_number=165, tag_tid=23, entry_tid=13085], [object_number=164, tag_tid=23, entry_tid=13084]]], state=connected] 2024-08-31T22:28:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=167, tag_tid=23, entry_tid=13087], [object_number=166, tag_tid=23, entry_tid=13086], [object_number=165, tag_tid=23, entry_tid=13085], [object_number=164, tag_tid=23, entry_tid=13084]]], state=connected] 2024-08-31T22:28:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=167, tag_tid=23, entry_tid=13087' 2024-08-31T22:28:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=167, tag_tid=23, entry_tid=13087' 2024-08-31T22:28:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=167, tag_tid=23, entry_tid=13087' '!=' 'object_number=164, tag_tid=23, entry_tid=13008' 2024-08-31T22:28:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=167, tag_tid=23, entry_tid=13087' = 'object_number=164, tag_tid=23, entry_tid=13008' 2024-08-31T22:28:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=167, tag_tid=23, entry_tid=13087' 2024-08-31T22:28:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=167, tag_tid=23, entry_tid=13087' 2024-08-31T22:28:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=167, tag_tid=23, entry_tid=13087' 2024-08-31T22:28:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13087 2024-08-31T22:28:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 13087 2024-08-31T22:28:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=166, tag_tid=23, entry_tid=13202], [object_number=165, tag_tid=23, entry_tid=13201], [object_number=164, tag_tid=23, entry_tid=13200], [object_number=167, tag_tid=23, entry_tid=13199]]], state=connected] 2024-08-31T22:28:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=166, tag_tid=23, entry_tid=13202], [object_number=165, tag_tid=23, entry_tid=13201], [object_number=164, tag_tid=23, entry_tid=13200], [object_number=167, tag_tid=23, entry_tid=13199]]], state=connected] 2024-08-31T22:28:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=166, tag_tid=23, entry_tid=13202' 2024-08-31T22:28:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=166, tag_tid=23, entry_tid=13202' 2024-08-31T22:28:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=166, tag_tid=23, entry_tid=13202' '!=' 'object_number=167, tag_tid=23, entry_tid=13087' 2024-08-31T22:28:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=166, tag_tid=23, entry_tid=13202' = 'object_number=167, tag_tid=23, entry_tid=13087' 2024-08-31T22:28:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=166, tag_tid=23, entry_tid=13202' 2024-08-31T22:28:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=166, tag_tid=23, entry_tid=13202' 2024-08-31T22:28:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=166, tag_tid=23, entry_tid=13202' 2024-08-31T22:28:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13202 2024-08-31T22:28:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 13202 2024-08-31T22:28:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:51.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:51.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:51.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=165, tag_tid=23, entry_tid=13221], [object_number=164, tag_tid=23, entry_tid=13220], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=165, tag_tid=23, entry_tid=13221], [object_number=164, tag_tid=23, entry_tid=13220], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=165, tag_tid=23, entry_tid=13221' 2024-08-31T22:28:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=165, tag_tid=23, entry_tid=13221' 2024-08-31T22:28:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=165, tag_tid=23, entry_tid=13221' '!=' 'object_number=166, tag_tid=23, entry_tid=13202' 2024-08-31T22:28:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:28:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=165, tag_tid=23, entry_tid=13221' = 'object_number=166, tag_tid=23, entry_tid=13202' 2024-08-31T22:28:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=165, tag_tid=23, entry_tid=13221' 2024-08-31T22:28:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=24 2024-08-31T22:28:51.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=23, entry_tid=13221' 2024-08-31T22:28:51.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:28:51.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:51.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=23 2024-08-31T22:28:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:28:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=165, tag_tid=23, entry_tid=13221' 2024-08-31T22:28:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:28:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:28:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=13221 2024-08-31T22:28:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 24 = 23 -a 1 -le 13221 2024-08-31T22:28:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:28:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:28:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=165, tag_tid=23, entry_tid=13221], [object_number=164, tag_tid=23, entry_tid=13220], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=165, tag_tid=23, entry_tid=13221], [object_number=164, tag_tid=23, entry_tid=13220], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:51.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:51.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=165, tag_tid=23, entry_tid=13221' 2024-08-31T22:28:51.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=165, tag_tid=23, entry_tid=13221' 2024-08-31T22:28:51.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=165, tag_tid=23, entry_tid=13221' '!=' 'object_number=165, tag_tid=23, entry_tid=13221' 2024-08-31T22:28:51.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:28:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.4 2024-08-31T22:28:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:28:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:28:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:28:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:28:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:28:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:28:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:28:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:28:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:28:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:28:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:28:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:28:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:28:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:28:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:28:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:28:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:28:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:28:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:28:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:28:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:28:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:28:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:28:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:28:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:28:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:28:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:28:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:28:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:28:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:28:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:28:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:28:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:28:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:28:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:28:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:28:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:28:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:28:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:28:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:28:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:28:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:28:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:28:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:28:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:28:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:28:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:28:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:28:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:28:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:28:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:28:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:28:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:28:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:28:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:28:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:28:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 41 2024-08-31T22:28:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 41 2024-08-31T22:28:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:28:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=165, tag_tid=24, entry_tid=1], [object_number=164, tag_tid=24, entry_tid=0], [object_number=167, tag_tid=23, entry_tid=13219], [object_number=166, tag_tid=23, entry_tid=13218]]], state=connected] 2024-08-31T22:28:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:28:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=165, tag_tid=24, entry_tid=1' -a 'object_number=165, tag_tid=24, entry_tid=1' = 'object_number=165, tag_tid=24, entry_tid=1' 2024-08-31T22:28:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-08-31T22:28:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:28:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap8 2024-08-31T22:28:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:28:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:28:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:28:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-08-31T22:28:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'test': 2024-08-31T22:28:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:28:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:28:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 15 2024-08-31T22:28:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a87b4c6a1b 2024-08-31T22:28:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a87b4c6a1b 2024-08-31T22:28:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:28:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:28:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:28:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:28:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:14:57 2024 2024-08-31T22:28:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:27:22 2024 2024-08-31T22:28:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:28:10 2024 2024-08-31T22:28:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:28:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a87b4c6a1b 2024-08-31T22:28:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:28:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:28:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:28:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:28:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n snap7-clean ']' 2024-08-31T22:28:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap7-clean 2024-08-31T22:28:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap7-clean 2024-08-31T22:28:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:28:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:28:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:28:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:28:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:28:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:28:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:28:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:28:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:28:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:28:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap8 2024-08-31T22:28:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap8 2024-08-31T22:28:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:28:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:28:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:28:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:28:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:28:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:28:56.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:28:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:28:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:28:58.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:28:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:28:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ clean_snap_name=snap8-clean 2024-08-31T22:28:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-08-31T22:28:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:28:58.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:58.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:58.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap8-clean 2024-08-31T22:28:58.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-08-31T22:28:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:28:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror test 2024-08-31T22:28:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:28:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:28:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:28:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:28:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=6 2024-08-31T22:28:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:28:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-08-31T22:28:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 21848064 bytes 2024-08-31T22:29:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 143217664 bytes 2024-08-31T22:29:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:29:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:29:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:29:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-08-31T22:29:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:29:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:29:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:29:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:29:06.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-08-31T22:29:06.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:06.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:06.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:29:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:29:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:29:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:29:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:29:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-08-31T22:29:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='test: 2024-08-31T22:29:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: global_id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:29:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+replaying 2024-08-31T22:29:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: replaying, {"bytes_per_second":2436753.7,"entries_behind_primary":0,"entries_per_second":157.57,"non_primary_position":{"entry_tid":1,"object_number":165,"tag_tid":24},"primary_position":{"entry_tid":1,"object_number":165,"tag_tid":24}} 2024-08-31T22:29:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: service: mirror.0 on smithi032 2024-08-31T22:29:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:28:55 2024-08-31T22:29:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: peer_sites: 2024-08-31T22:29:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: name: cluster2 2024-08-31T22:29:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+stopped 2024-08-31T22:29:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: local image is primary 2024-08-31T22:29:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:29:05' 2024-08-31T22:29:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ state: .*up+replaying' 2024-08-31T22:29:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: state: up+replaying 2024-08-31T22:29:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ description: .*' 2024-08-31T22:29:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: description: replaying, {"bytes_per_second":2436753.7,"entries_behind_primary":0,"entries_per_second":157.57,"non_primary_position":{"entry_tid":1,"object_number":165,"tag_tid":24},"primary_position":{"entry_tid":1,"object_number":165,"tag_tid":24}} 2024-08-31T22:29:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:29:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:29:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^up+' 2024-08-31T22:29:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:29:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep 'service: *mirror..* on ' 2024-08-31T22:29:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: service: mirror.0 on smithi032 2024-08-31T22:29:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update 2024-08-31T22:29:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-08-31T22:29:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:28:55' 2024-08-31T22:29:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-08-31 22:28:55' 2024-08-31T22:29:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:29:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'prev_last_update=2024-08-31 22:28:55' 2024-08-31T22:29:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:29:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:29:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-08-31T22:29:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='OKOKOK142574264mirror.0smithi03219.1.1-244-g3091dfc0trueOK42614267mirror.1smithi03219.1.1-244-g3091dfc0falseOKtestdd2c705e-25cf-4565-981f-d48de48eee84up+replayingreplaying, {"bytes_per_second":2436753.7,"entries_behind_primary":0,"entries_per_second":157.57,"non_primary_position":{"entry_tid":1,"object_number":165,"tag_tid":24},"primary_position":{"entry_tid":1,"object_number":165,"tag_tid":24}}42574264mirror.0smithi0322024-08-31 22:28:55cluster24ca32b00-2164-486e-837e-e93795a16af1up+stoppedlocal image is primary2024-08-31 22:29:05' 2024-08-31T22:29:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update state 2024-08-31T22:29:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-08-31T22:29:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:28:55' 2024-08-31T22:29:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-08-31T22:29:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ state=up+replaying 2024-08-31T22:29:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:29:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep up+replaying 2024-08-31T22:29:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:29:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap9 2024-08-31T22:29:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap9 2024-08-31T22:29:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:29:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap9 2024-08-31T22:29:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:29:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-08-31T22:29:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-08-31T22:29:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:29:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:29:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:29:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror test started 2024-08-31T22:29:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:29:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:29:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:29:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-08-31T22:29:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:29:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:29:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:29:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:29:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:29:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:29:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:29:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-08-31T22:29:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:29:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:29:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:29:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:29:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:29:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:29:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:29:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:29:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:29:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:29:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:29:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:29:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:29:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:29:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:29:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:29:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-08-31T22:29:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:29:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:29:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:29:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:29:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:29:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:29:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:29:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:29:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:29:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:29:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:29:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:29:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:29:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:29:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:29:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:29:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:38.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:38.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:38.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:38.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:38.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:38.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 42 2024-08-31T22:29:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=172, tag_tid=26, entry_tid=5676], [object_number=175, tag_tid=26, entry_tid=5675], [object_number=170, tag_tid=26, entry_tid=5674], [object_number=173, tag_tid=26, entry_tid=5673]]], state=connected] 2024-08-31T22:29:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:38.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 42 2024-08-31T22:29:38.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:38.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=172, tag_tid=26, entry_tid=5676], [object_number=175, tag_tid=26, entry_tid=5675], [object_number=170, tag_tid=26, entry_tid=5674], [object_number=173, tag_tid=26, entry_tid=5673]]], state=connected] 2024-08-31T22:29:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=172, tag_tid=26, entry_tid=5676' 2024-08-31T22:29:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=172, tag_tid=26, entry_tid=5676' 2024-08-31T22:29:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=172, tag_tid=26, entry_tid=5676' '!=' '' 2024-08-31T22:29:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=172, tag_tid=26, entry_tid=5676' = '' 2024-08-31T22:29:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=172, tag_tid=26, entry_tid=5676' 2024-08-31T22:29:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=172, tag_tid=26, entry_tid=5676' 2024-08-31T22:29:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=172, tag_tid=26, entry_tid=5676' 2024-08-31T22:29:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=5676 2024-08-31T22:29:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 5676 2024-08-31T22:29:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 42 2024-08-31T22:29:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=5763], [object_number=170, tag_tid=26, entry_tid=5762], [object_number=173, tag_tid=26, entry_tid=5761], [object_number=172, tag_tid=26, entry_tid=5760]]], state=connected] 2024-08-31T22:29:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 42 2024-08-31T22:29:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=5763], [object_number=170, tag_tid=26, entry_tid=5762], [object_number=173, tag_tid=26, entry_tid=5761], [object_number=172, tag_tid=26, entry_tid=5760]]], state=connected] 2024-08-31T22:29:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=175, tag_tid=26, entry_tid=5763' 2024-08-31T22:29:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=175, tag_tid=26, entry_tid=5763' 2024-08-31T22:29:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=5763' '!=' 'object_number=172, tag_tid=26, entry_tid=5676' 2024-08-31T22:29:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=5763' = 'object_number=172, tag_tid=26, entry_tid=5676' 2024-08-31T22:29:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=175, tag_tid=26, entry_tid=5763' 2024-08-31T22:29:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=5763' 2024-08-31T22:29:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=5763' 2024-08-31T22:29:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=5763 2024-08-31T22:29:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 5763 2024-08-31T22:29:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=5830], [object_number=173, tag_tid=26, entry_tid=5829], [object_number=172, tag_tid=26, entry_tid=5828], [object_number=175, tag_tid=26, entry_tid=5827]]], state=connected] 2024-08-31T22:29:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=5830], [object_number=173, tag_tid=26, entry_tid=5829], [object_number=172, tag_tid=26, entry_tid=5828], [object_number=175, tag_tid=26, entry_tid=5827]]], state=connected] 2024-08-31T22:29:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=174, tag_tid=26, entry_tid=5830' 2024-08-31T22:29:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=174, tag_tid=26, entry_tid=5830' 2024-08-31T22:29:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=5830' '!=' 'object_number=175, tag_tid=26, entry_tid=5763' 2024-08-31T22:29:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=5830' = 'object_number=175, tag_tid=26, entry_tid=5763' 2024-08-31T22:29:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=174, tag_tid=26, entry_tid=5830' 2024-08-31T22:29:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:39.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=5830' 2024-08-31T22:29:39.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:39.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=5830' 2024-08-31T22:29:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=5830 2024-08-31T22:29:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 5830 2024-08-31T22:29:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=5911], [object_number=174, tag_tid=26, entry_tid=5910], [object_number=173, tag_tid=26, entry_tid=5909], [object_number=172, tag_tid=26, entry_tid=5908]]], state=connected] 2024-08-31T22:29:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=5911], [object_number=174, tag_tid=26, entry_tid=5910], [object_number=173, tag_tid=26, entry_tid=5909], [object_number=172, tag_tid=26, entry_tid=5908]]], state=connected] 2024-08-31T22:29:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=175, tag_tid=26, entry_tid=5911' 2024-08-31T22:29:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=175, tag_tid=26, entry_tid=5911' 2024-08-31T22:29:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=5911' '!=' 'object_number=174, tag_tid=26, entry_tid=5830' 2024-08-31T22:29:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=5911' = 'object_number=174, tag_tid=26, entry_tid=5830' 2024-08-31T22:29:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=175, tag_tid=26, entry_tid=5911' 2024-08-31T22:29:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=5911' 2024-08-31T22:29:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=5911' 2024-08-31T22:29:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=5911 2024-08-31T22:29:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 5911 2024-08-31T22:29:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=173, tag_tid=26, entry_tid=5993], [object_number=172, tag_tid=26, entry_tid=5992], [object_number=175, tag_tid=26, entry_tid=5991], [object_number=174, tag_tid=26, entry_tid=5990]]], state=connected] 2024-08-31T22:29:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:40.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:40.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:40.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:40.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:40.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=173, tag_tid=26, entry_tid=5993], [object_number=172, tag_tid=26, entry_tid=5992], [object_number=175, tag_tid=26, entry_tid=5991], [object_number=174, tag_tid=26, entry_tid=5990]]], state=connected] 2024-08-31T22:29:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=173, tag_tid=26, entry_tid=5993' 2024-08-31T22:29:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=173, tag_tid=26, entry_tid=5993' 2024-08-31T22:29:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=173, tag_tid=26, entry_tid=5993' '!=' 'object_number=175, tag_tid=26, entry_tid=5911' 2024-08-31T22:29:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=173, tag_tid=26, entry_tid=5993' = 'object_number=175, tag_tid=26, entry_tid=5911' 2024-08-31T22:29:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=173, tag_tid=26, entry_tid=5993' 2024-08-31T22:29:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=173, tag_tid=26, entry_tid=5993' 2024-08-31T22:29:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=173, tag_tid=26, entry_tid=5993' 2024-08-31T22:29:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=5993 2024-08-31T22:29:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 5993 2024-08-31T22:29:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=6091], [object_number=174, tag_tid=26, entry_tid=6090], [object_number=173, tag_tid=26, entry_tid=6089], [object_number=172, tag_tid=26, entry_tid=6088]]], state=connected] 2024-08-31T22:29:41.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=6091], [object_number=174, tag_tid=26, entry_tid=6090], [object_number=173, tag_tid=26, entry_tid=6089], [object_number=172, tag_tid=26, entry_tid=6088]]], state=connected] 2024-08-31T22:29:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=175, tag_tid=26, entry_tid=6091' 2024-08-31T22:29:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=175, tag_tid=26, entry_tid=6091' 2024-08-31T22:29:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=6091' '!=' 'object_number=173, tag_tid=26, entry_tid=5993' 2024-08-31T22:29:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=6091' = 'object_number=173, tag_tid=26, entry_tid=5993' 2024-08-31T22:29:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=175, tag_tid=26, entry_tid=6091' 2024-08-31T22:29:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=6091' 2024-08-31T22:29:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=6091' 2024-08-31T22:29:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6091 2024-08-31T22:29:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 6091 2024-08-31T22:29:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:41.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:41.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:41.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:41.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:41.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:41.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:41.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:41.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=172, tag_tid=26, entry_tid=6188], [object_number=175, tag_tid=26, entry_tid=6187], [object_number=174, tag_tid=26, entry_tid=6186], [object_number=173, tag_tid=26, entry_tid=6185]]], state=connected] 2024-08-31T22:29:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=172, tag_tid=26, entry_tid=6188], [object_number=175, tag_tid=26, entry_tid=6187], [object_number=174, tag_tid=26, entry_tid=6186], [object_number=173, tag_tid=26, entry_tid=6185]]], state=connected] 2024-08-31T22:29:41.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=172, tag_tid=26, entry_tid=6188' 2024-08-31T22:29:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=172, tag_tid=26, entry_tid=6188' 2024-08-31T22:29:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=172, tag_tid=26, entry_tid=6188' '!=' 'object_number=175, tag_tid=26, entry_tid=6091' 2024-08-31T22:29:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=172, tag_tid=26, entry_tid=6188' = 'object_number=175, tag_tid=26, entry_tid=6091' 2024-08-31T22:29:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=172, tag_tid=26, entry_tid=6188' 2024-08-31T22:29:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=172, tag_tid=26, entry_tid=6188' 2024-08-31T22:29:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=172, tag_tid=26, entry_tid=6188' 2024-08-31T22:29:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6188 2024-08-31T22:29:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 6188 2024-08-31T22:29:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=172, tag_tid=26, entry_tid=6288], [object_number=175, tag_tid=26, entry_tid=6287], [object_number=174, tag_tid=26, entry_tid=6286], [object_number=173, tag_tid=26, entry_tid=6285]]], state=connected] 2024-08-31T22:29:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=172, tag_tid=26, entry_tid=6288], [object_number=175, tag_tid=26, entry_tid=6287], [object_number=174, tag_tid=26, entry_tid=6286], [object_number=173, tag_tid=26, entry_tid=6285]]], state=connected] 2024-08-31T22:29:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=172, tag_tid=26, entry_tid=6288' 2024-08-31T22:29:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=172, tag_tid=26, entry_tid=6288' 2024-08-31T22:29:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=172, tag_tid=26, entry_tid=6288' '!=' 'object_number=172, tag_tid=26, entry_tid=6188' 2024-08-31T22:29:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=172, tag_tid=26, entry_tid=6288' = 'object_number=172, tag_tid=26, entry_tid=6188' 2024-08-31T22:29:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=172, tag_tid=26, entry_tid=6288' 2024-08-31T22:29:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=172, tag_tid=26, entry_tid=6288' 2024-08-31T22:29:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:42.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:42.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:42.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:42.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:42.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=172, tag_tid=26, entry_tid=6288' 2024-08-31T22:29:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6288 2024-08-31T22:29:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 6288 2024-08-31T22:29:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:42.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=6375], [object_number=174, tag_tid=26, entry_tid=6374], [object_number=173, tag_tid=26, entry_tid=6373], [object_number=172, tag_tid=26, entry_tid=6372]]], state=connected] 2024-08-31T22:29:42.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:42.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:42.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:42.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:42.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:42.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:42.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:42.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:42.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=6375], [object_number=174, tag_tid=26, entry_tid=6374], [object_number=173, tag_tid=26, entry_tid=6373], [object_number=172, tag_tid=26, entry_tid=6372]]], state=connected] 2024-08-31T22:29:42.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:42.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=175, tag_tid=26, entry_tid=6375' 2024-08-31T22:29:42.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=175, tag_tid=26, entry_tid=6375' 2024-08-31T22:29:42.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=6375' '!=' 'object_number=172, tag_tid=26, entry_tid=6288' 2024-08-31T22:29:42.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:42.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=6375' = 'object_number=172, tag_tid=26, entry_tid=6288' 2024-08-31T22:29:42.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=175, tag_tid=26, entry_tid=6375' 2024-08-31T22:29:42.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:42.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:42.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=6375' 2024-08-31T22:29:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:42.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=6375' 2024-08-31T22:29:42.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:42.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6375 2024-08-31T22:29:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 6375 2024-08-31T22:29:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=172, tag_tid=26, entry_tid=6456], [object_number=175, tag_tid=26, entry_tid=6455], [object_number=174, tag_tid=26, entry_tid=6454], [object_number=173, tag_tid=26, entry_tid=6453]]], state=connected] 2024-08-31T22:29:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=172, tag_tid=26, entry_tid=6456], [object_number=175, tag_tid=26, entry_tid=6455], [object_number=174, tag_tid=26, entry_tid=6454], [object_number=173, tag_tid=26, entry_tid=6453]]], state=connected] 2024-08-31T22:29:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=172, tag_tid=26, entry_tid=6456' 2024-08-31T22:29:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=172, tag_tid=26, entry_tid=6456' 2024-08-31T22:29:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=172, tag_tid=26, entry_tid=6456' '!=' 'object_number=175, tag_tid=26, entry_tid=6375' 2024-08-31T22:29:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=172, tag_tid=26, entry_tid=6456' = 'object_number=175, tag_tid=26, entry_tid=6375' 2024-08-31T22:29:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=172, tag_tid=26, entry_tid=6456' 2024-08-31T22:29:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:43.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=172, tag_tid=26, entry_tid=6456' 2024-08-31T22:29:43.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:43.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:43.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:43.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:43.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:43.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=172, tag_tid=26, entry_tid=6456' 2024-08-31T22:29:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6456 2024-08-31T22:29:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 6456 2024-08-31T22:29:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=6535], [object_number=174, tag_tid=26, entry_tid=6534], [object_number=173, tag_tid=26, entry_tid=6533], [object_number=172, tag_tid=26, entry_tid=6532]]], state=connected] 2024-08-31T22:29:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=6535], [object_number=174, tag_tid=26, entry_tid=6534], [object_number=173, tag_tid=26, entry_tid=6533], [object_number=172, tag_tid=26, entry_tid=6532]]], state=connected] 2024-08-31T22:29:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=175, tag_tid=26, entry_tid=6535' 2024-08-31T22:29:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=175, tag_tid=26, entry_tid=6535' 2024-08-31T22:29:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=6535' '!=' 'object_number=172, tag_tid=26, entry_tid=6456' 2024-08-31T22:29:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=6535' = 'object_number=172, tag_tid=26, entry_tid=6456' 2024-08-31T22:29:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=175, tag_tid=26, entry_tid=6535' 2024-08-31T22:29:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=6535' 2024-08-31T22:29:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=6535' 2024-08-31T22:29:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6535 2024-08-31T22:29:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 6535 2024-08-31T22:29:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:44.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:44.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=173, tag_tid=26, entry_tid=6621], [object_number=172, tag_tid=26, entry_tid=6620], [object_number=175, tag_tid=26, entry_tid=6619], [object_number=174, tag_tid=26, entry_tid=6618]]], state=connected] 2024-08-31T22:29:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=173, tag_tid=26, entry_tid=6621], [object_number=172, tag_tid=26, entry_tid=6620], [object_number=175, tag_tid=26, entry_tid=6619], [object_number=174, tag_tid=26, entry_tid=6618]]], state=connected] 2024-08-31T22:29:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=173, tag_tid=26, entry_tid=6621' 2024-08-31T22:29:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=173, tag_tid=26, entry_tid=6621' 2024-08-31T22:29:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=173, tag_tid=26, entry_tid=6621' '!=' 'object_number=175, tag_tid=26, entry_tid=6535' 2024-08-31T22:29:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=173, tag_tid=26, entry_tid=6621' = 'object_number=175, tag_tid=26, entry_tid=6535' 2024-08-31T22:29:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=173, tag_tid=26, entry_tid=6621' 2024-08-31T22:29:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=173, tag_tid=26, entry_tid=6621' 2024-08-31T22:29:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:44.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:44.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:44.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:44.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=173, tag_tid=26, entry_tid=6621' 2024-08-31T22:29:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6621 2024-08-31T22:29:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 6621 2024-08-31T22:29:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:44.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=6707], [object_number=174, tag_tid=26, entry_tid=6706], [object_number=173, tag_tid=26, entry_tid=6705], [object_number=172, tag_tid=26, entry_tid=6704]]], state=connected] 2024-08-31T22:29:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=6707], [object_number=174, tag_tid=26, entry_tid=6706], [object_number=173, tag_tid=26, entry_tid=6705], [object_number=172, tag_tid=26, entry_tid=6704]]], state=connected] 2024-08-31T22:29:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=175, tag_tid=26, entry_tid=6707' 2024-08-31T22:29:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=175, tag_tid=26, entry_tid=6707' 2024-08-31T22:29:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=6707' '!=' 'object_number=173, tag_tid=26, entry_tid=6621' 2024-08-31T22:29:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=6707' = 'object_number=173, tag_tid=26, entry_tid=6621' 2024-08-31T22:29:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=175, tag_tid=26, entry_tid=6707' 2024-08-31T22:29:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=6707' 2024-08-31T22:29:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=6707' 2024-08-31T22:29:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6707 2024-08-31T22:29:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 6707 2024-08-31T22:29:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=173, tag_tid=26, entry_tid=6797], [object_number=172, tag_tid=26, entry_tid=6796], [object_number=175, tag_tid=26, entry_tid=6795], [object_number=174, tag_tid=26, entry_tid=6794]]], state=connected] 2024-08-31T22:29:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:45.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=173, tag_tid=26, entry_tid=6797], [object_number=172, tag_tid=26, entry_tid=6796], [object_number=175, tag_tid=26, entry_tid=6795], [object_number=174, tag_tid=26, entry_tid=6794]]], state=connected] 2024-08-31T22:29:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=173, tag_tid=26, entry_tid=6797' 2024-08-31T22:29:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=173, tag_tid=26, entry_tid=6797' 2024-08-31T22:29:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=173, tag_tid=26, entry_tid=6797' '!=' 'object_number=175, tag_tid=26, entry_tid=6707' 2024-08-31T22:29:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=173, tag_tid=26, entry_tid=6797' = 'object_number=175, tag_tid=26, entry_tid=6707' 2024-08-31T22:29:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=173, tag_tid=26, entry_tid=6797' 2024-08-31T22:29:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=173, tag_tid=26, entry_tid=6797' 2024-08-31T22:29:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=173, tag_tid=26, entry_tid=6797' 2024-08-31T22:29:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6797 2024-08-31T22:29:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 6797 2024-08-31T22:29:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=6890], [object_number=173, tag_tid=26, entry_tid=6889], [object_number=172, tag_tid=26, entry_tid=6888], [object_number=175, tag_tid=26, entry_tid=6887]]], state=connected] 2024-08-31T22:29:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:45.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=6890], [object_number=173, tag_tid=26, entry_tid=6889], [object_number=172, tag_tid=26, entry_tid=6888], [object_number=175, tag_tid=26, entry_tid=6887]]], state=connected] 2024-08-31T22:29:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=174, tag_tid=26, entry_tid=6890' 2024-08-31T22:29:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=174, tag_tid=26, entry_tid=6890' 2024-08-31T22:29:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=6890' '!=' 'object_number=173, tag_tid=26, entry_tid=6797' 2024-08-31T22:29:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=6890' = 'object_number=173, tag_tid=26, entry_tid=6797' 2024-08-31T22:29:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=174, tag_tid=26, entry_tid=6890' 2024-08-31T22:29:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=6890' 2024-08-31T22:29:45.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:45.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=6890' 2024-08-31T22:29:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6890 2024-08-31T22:29:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 6890 2024-08-31T22:29:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=6986], [object_number=173, tag_tid=26, entry_tid=6985], [object_number=172, tag_tid=26, entry_tid=6984], [object_number=175, tag_tid=26, entry_tid=6983]]], state=connected] 2024-08-31T22:29:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=6986], [object_number=173, tag_tid=26, entry_tid=6985], [object_number=172, tag_tid=26, entry_tid=6984], [object_number=175, tag_tid=26, entry_tid=6983]]], state=connected] 2024-08-31T22:29:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=174, tag_tid=26, entry_tid=6986' 2024-08-31T22:29:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=174, tag_tid=26, entry_tid=6986' 2024-08-31T22:29:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=6986' '!=' 'object_number=174, tag_tid=26, entry_tid=6890' 2024-08-31T22:29:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=6986' = 'object_number=174, tag_tid=26, entry_tid=6890' 2024-08-31T22:29:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=174, tag_tid=26, entry_tid=6986' 2024-08-31T22:29:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=6986' 2024-08-31T22:29:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=6986' 2024-08-31T22:29:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6986 2024-08-31T22:29:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 6986 2024-08-31T22:29:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=7074], [object_number=173, tag_tid=26, entry_tid=7073], [object_number=172, tag_tid=26, entry_tid=7072], [object_number=175, tag_tid=26, entry_tid=7071]]], state=connected] 2024-08-31T22:29:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:46.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=7074], [object_number=173, tag_tid=26, entry_tid=7073], [object_number=172, tag_tid=26, entry_tid=7072], [object_number=175, tag_tid=26, entry_tid=7071]]], state=connected] 2024-08-31T22:29:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=174, tag_tid=26, entry_tid=7074' 2024-08-31T22:29:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=174, tag_tid=26, entry_tid=7074' 2024-08-31T22:29:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=7074' '!=' 'object_number=174, tag_tid=26, entry_tid=6986' 2024-08-31T22:29:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=7074' = 'object_number=174, tag_tid=26, entry_tid=6986' 2024-08-31T22:29:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=174, tag_tid=26, entry_tid=7074' 2024-08-31T22:29:46.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:46.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:46.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:46.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=7074' 2024-08-31T22:29:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:46.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:46.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:46.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=7074' 2024-08-31T22:29:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7074 2024-08-31T22:29:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 7074 2024-08-31T22:29:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=7163], [object_number=174, tag_tid=26, entry_tid=7162], [object_number=173, tag_tid=26, entry_tid=7161], [object_number=172, tag_tid=26, entry_tid=7160]]], state=connected] 2024-08-31T22:29:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:47.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:47.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:47.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=7163], [object_number=174, tag_tid=26, entry_tid=7162], [object_number=173, tag_tid=26, entry_tid=7161], [object_number=172, tag_tid=26, entry_tid=7160]]], state=connected] 2024-08-31T22:29:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=175, tag_tid=26, entry_tid=7163' 2024-08-31T22:29:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=175, tag_tid=26, entry_tid=7163' 2024-08-31T22:29:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=7163' '!=' 'object_number=174, tag_tid=26, entry_tid=7074' 2024-08-31T22:29:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=7163' = 'object_number=174, tag_tid=26, entry_tid=7074' 2024-08-31T22:29:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=175, tag_tid=26, entry_tid=7163' 2024-08-31T22:29:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=7163' 2024-08-31T22:29:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=7163' 2024-08-31T22:29:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7163 2024-08-31T22:29:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 7163 2024-08-31T22:29:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=7246], [object_number=173, tag_tid=26, entry_tid=7245], [object_number=172, tag_tid=26, entry_tid=7244], [object_number=175, tag_tid=26, entry_tid=7243]]], state=connected] 2024-08-31T22:29:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=7246], [object_number=173, tag_tid=26, entry_tid=7245], [object_number=172, tag_tid=26, entry_tid=7244], [object_number=175, tag_tid=26, entry_tid=7243]]], state=connected] 2024-08-31T22:29:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=174, tag_tid=26, entry_tid=7246' 2024-08-31T22:29:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=174, tag_tid=26, entry_tid=7246' 2024-08-31T22:29:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=7246' '!=' 'object_number=175, tag_tid=26, entry_tid=7163' 2024-08-31T22:29:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=7246' = 'object_number=175, tag_tid=26, entry_tid=7163' 2024-08-31T22:29:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=174, tag_tid=26, entry_tid=7246' 2024-08-31T22:29:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=7246' 2024-08-31T22:29:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=7246' 2024-08-31T22:29:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7246 2024-08-31T22:29:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 7246 2024-08-31T22:29:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=7322], [object_number=173, tag_tid=26, entry_tid=7321], [object_number=172, tag_tid=26, entry_tid=7320], [object_number=175, tag_tid=26, entry_tid=7319]]], state=connected] 2024-08-31T22:29:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=7322], [object_number=173, tag_tid=26, entry_tid=7321], [object_number=172, tag_tid=26, entry_tid=7320], [object_number=175, tag_tid=26, entry_tid=7319]]], state=connected] 2024-08-31T22:29:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=174, tag_tid=26, entry_tid=7322' 2024-08-31T22:29:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=174, tag_tid=26, entry_tid=7322' 2024-08-31T22:29:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=7322' '!=' 'object_number=174, tag_tid=26, entry_tid=7246' 2024-08-31T22:29:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=7322' = 'object_number=174, tag_tid=26, entry_tid=7246' 2024-08-31T22:29:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=174, tag_tid=26, entry_tid=7322' 2024-08-31T22:29:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=7322' 2024-08-31T22:29:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=7322' 2024-08-31T22:29:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7322 2024-08-31T22:29:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 7322 2024-08-31T22:29:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=173, tag_tid=26, entry_tid=7413], [object_number=172, tag_tid=26, entry_tid=7412], [object_number=175, tag_tid=26, entry_tid=7411], [object_number=174, tag_tid=26, entry_tid=7410]]], state=connected] 2024-08-31T22:29:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=173, tag_tid=26, entry_tid=7413], [object_number=172, tag_tid=26, entry_tid=7412], [object_number=175, tag_tid=26, entry_tid=7411], [object_number=174, tag_tid=26, entry_tid=7410]]], state=connected] 2024-08-31T22:29:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=173, tag_tid=26, entry_tid=7413' 2024-08-31T22:29:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=173, tag_tid=26, entry_tid=7413' 2024-08-31T22:29:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=173, tag_tid=26, entry_tid=7413' '!=' 'object_number=174, tag_tid=26, entry_tid=7322' 2024-08-31T22:29:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=173, tag_tid=26, entry_tid=7413' = 'object_number=174, tag_tid=26, entry_tid=7322' 2024-08-31T22:29:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=173, tag_tid=26, entry_tid=7413' 2024-08-31T22:29:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=173, tag_tid=26, entry_tid=7413' 2024-08-31T22:29:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=173, tag_tid=26, entry_tid=7413' 2024-08-31T22:29:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7413 2024-08-31T22:29:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 7413 2024-08-31T22:29:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=7510], [object_number=173, tag_tid=26, entry_tid=7509], [object_number=172, tag_tid=26, entry_tid=7508], [object_number=175, tag_tid=26, entry_tid=7507]]], state=connected] 2024-08-31T22:29:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=7510], [object_number=173, tag_tid=26, entry_tid=7509], [object_number=172, tag_tid=26, entry_tid=7508], [object_number=175, tag_tid=26, entry_tid=7507]]], state=connected] 2024-08-31T22:29:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:49.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=174, tag_tid=26, entry_tid=7510' 2024-08-31T22:29:49.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=174, tag_tid=26, entry_tid=7510' 2024-08-31T22:29:49.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=7510' '!=' 'object_number=173, tag_tid=26, entry_tid=7413' 2024-08-31T22:29:49.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:49.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=7510' = 'object_number=173, tag_tid=26, entry_tid=7413' 2024-08-31T22:29:49.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=174, tag_tid=26, entry_tid=7510' 2024-08-31T22:29:49.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:49.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:49.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:49.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:49.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=7510' 2024-08-31T22:29:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:49.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=7510' 2024-08-31T22:29:49.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:49.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7510 2024-08-31T22:29:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 7510 2024-08-31T22:29:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:49.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:49.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:49.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=172, tag_tid=26, entry_tid=7596], [object_number=175, tag_tid=26, entry_tid=7595], [object_number=174, tag_tid=26, entry_tid=7594], [object_number=173, tag_tid=26, entry_tid=7593]]], state=connected] 2024-08-31T22:29:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=172, tag_tid=26, entry_tid=7596], [object_number=175, tag_tid=26, entry_tid=7595], [object_number=174, tag_tid=26, entry_tid=7594], [object_number=173, tag_tid=26, entry_tid=7593]]], state=connected] 2024-08-31T22:29:49.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=172, tag_tid=26, entry_tid=7596' 2024-08-31T22:29:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=172, tag_tid=26, entry_tid=7596' 2024-08-31T22:29:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=172, tag_tid=26, entry_tid=7596' '!=' 'object_number=174, tag_tid=26, entry_tid=7510' 2024-08-31T22:29:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=172, tag_tid=26, entry_tid=7596' = 'object_number=174, tag_tid=26, entry_tid=7510' 2024-08-31T22:29:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=172, tag_tid=26, entry_tid=7596' 2024-08-31T22:29:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=172, tag_tid=26, entry_tid=7596' 2024-08-31T22:29:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=172, tag_tid=26, entry_tid=7596' 2024-08-31T22:29:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7596 2024-08-31T22:29:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 7596 2024-08-31T22:29:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=7719], [object_number=174, tag_tid=26, entry_tid=7718], [object_number=173, tag_tid=26, entry_tid=7717], [object_number=172, tag_tid=26, entry_tid=7716]]], state=connected] 2024-08-31T22:29:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=7719], [object_number=174, tag_tid=26, entry_tid=7718], [object_number=173, tag_tid=26, entry_tid=7717], [object_number=172, tag_tid=26, entry_tid=7716]]], state=connected] 2024-08-31T22:29:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=175, tag_tid=26, entry_tid=7719' 2024-08-31T22:29:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=175, tag_tid=26, entry_tid=7719' 2024-08-31T22:29:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=7719' '!=' 'object_number=172, tag_tid=26, entry_tid=7596' 2024-08-31T22:29:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=7719' = 'object_number=172, tag_tid=26, entry_tid=7596' 2024-08-31T22:29:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=175, tag_tid=26, entry_tid=7719' 2024-08-31T22:29:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=7719' 2024-08-31T22:29:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=7719' 2024-08-31T22:29:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7719 2024-08-31T22:29:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 7719 2024-08-31T22:29:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:50.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=172, tag_tid=26, entry_tid=7796], [object_number=175, tag_tid=26, entry_tid=7795], [object_number=174, tag_tid=26, entry_tid=7794], [object_number=173, tag_tid=26, entry_tid=7793]]], state=connected] 2024-08-31T22:29:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=172, tag_tid=26, entry_tid=7796], [object_number=175, tag_tid=26, entry_tid=7795], [object_number=174, tag_tid=26, entry_tid=7794], [object_number=173, tag_tid=26, entry_tid=7793]]], state=connected] 2024-08-31T22:29:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=172, tag_tid=26, entry_tid=7796' 2024-08-31T22:29:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=172, tag_tid=26, entry_tid=7796' 2024-08-31T22:29:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=172, tag_tid=26, entry_tid=7796' '!=' 'object_number=175, tag_tid=26, entry_tid=7719' 2024-08-31T22:29:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=172, tag_tid=26, entry_tid=7796' = 'object_number=175, tag_tid=26, entry_tid=7719' 2024-08-31T22:29:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=172, tag_tid=26, entry_tid=7796' 2024-08-31T22:29:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=172, tag_tid=26, entry_tid=7796' 2024-08-31T22:29:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=172, tag_tid=26, entry_tid=7796' 2024-08-31T22:29:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7796 2024-08-31T22:29:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 7796 2024-08-31T22:29:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:51.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:51.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:51.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:51.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:51.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:51.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:51.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:51.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:51.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:51.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:51.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:51.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:51.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:51.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:51.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:51.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=7883], [object_number=174, tag_tid=26, entry_tid=7882], [object_number=173, tag_tid=26, entry_tid=7881], [object_number=172, tag_tid=26, entry_tid=7880]]], state=connected] 2024-08-31T22:29:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=7883], [object_number=174, tag_tid=26, entry_tid=7882], [object_number=173, tag_tid=26, entry_tid=7881], [object_number=172, tag_tid=26, entry_tid=7880]]], state=connected] 2024-08-31T22:29:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=175, tag_tid=26, entry_tid=7883' 2024-08-31T22:29:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=175, tag_tid=26, entry_tid=7883' 2024-08-31T22:29:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=7883' '!=' 'object_number=172, tag_tid=26, entry_tid=7796' 2024-08-31T22:29:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=7883' = 'object_number=172, tag_tid=26, entry_tid=7796' 2024-08-31T22:29:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=175, tag_tid=26, entry_tid=7883' 2024-08-31T22:29:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=7883' 2024-08-31T22:29:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=7883' 2024-08-31T22:29:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7883 2024-08-31T22:29:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 7883 2024-08-31T22:29:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:51.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:51.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:51.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:51.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:51.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:51.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=173, tag_tid=26, entry_tid=7969], [object_number=172, tag_tid=26, entry_tid=7968], [object_number=175, tag_tid=26, entry_tid=7967], [object_number=174, tag_tid=26, entry_tid=7966]]], state=connected] 2024-08-31T22:29:52.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=173, tag_tid=26, entry_tid=7969], [object_number=172, tag_tid=26, entry_tid=7968], [object_number=175, tag_tid=26, entry_tid=7967], [object_number=174, tag_tid=26, entry_tid=7966]]], state=connected] 2024-08-31T22:29:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=173, tag_tid=26, entry_tid=7969' 2024-08-31T22:29:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=173, tag_tid=26, entry_tid=7969' 2024-08-31T22:29:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=173, tag_tid=26, entry_tid=7969' '!=' 'object_number=175, tag_tid=26, entry_tid=7883' 2024-08-31T22:29:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=173, tag_tid=26, entry_tid=7969' = 'object_number=175, tag_tid=26, entry_tid=7883' 2024-08-31T22:29:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=173, tag_tid=26, entry_tid=7969' 2024-08-31T22:29:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=173, tag_tid=26, entry_tid=7969' 2024-08-31T22:29:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=173, tag_tid=26, entry_tid=7969' 2024-08-31T22:29:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7969 2024-08-31T22:29:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 7969 2024-08-31T22:29:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:52.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=173, tag_tid=26, entry_tid=8053], [object_number=172, tag_tid=26, entry_tid=8052], [object_number=175, tag_tid=26, entry_tid=8051], [object_number=174, tag_tid=26, entry_tid=8050]]], state=connected] 2024-08-31T22:29:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:52.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:52.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:52.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:52.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=173, tag_tid=26, entry_tid=8053], [object_number=172, tag_tid=26, entry_tid=8052], [object_number=175, tag_tid=26, entry_tid=8051], [object_number=174, tag_tid=26, entry_tid=8050]]], state=connected] 2024-08-31T22:29:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=173, tag_tid=26, entry_tid=8053' 2024-08-31T22:29:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=173, tag_tid=26, entry_tid=8053' 2024-08-31T22:29:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=173, tag_tid=26, entry_tid=8053' '!=' 'object_number=173, tag_tid=26, entry_tid=7969' 2024-08-31T22:29:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=173, tag_tid=26, entry_tid=8053' = 'object_number=173, tag_tid=26, entry_tid=7969' 2024-08-31T22:29:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=173, tag_tid=26, entry_tid=8053' 2024-08-31T22:29:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=173, tag_tid=26, entry_tid=8053' 2024-08-31T22:29:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=173, tag_tid=26, entry_tid=8053' 2024-08-31T22:29:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8053 2024-08-31T22:29:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 8053 2024-08-31T22:29:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:53.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:53.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:53.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=8146], [object_number=173, tag_tid=26, entry_tid=8145], [object_number=172, tag_tid=26, entry_tid=8144], [object_number=175, tag_tid=26, entry_tid=8143]]], state=connected] 2024-08-31T22:29:53.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:53.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:53.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:53.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:53.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:53.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:53.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:53.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:53.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:53.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=8146], [object_number=173, tag_tid=26, entry_tid=8145], [object_number=172, tag_tid=26, entry_tid=8144], [object_number=175, tag_tid=26, entry_tid=8143]]], state=connected] 2024-08-31T22:29:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:53.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=174, tag_tid=26, entry_tid=8146' 2024-08-31T22:29:53.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=174, tag_tid=26, entry_tid=8146' 2024-08-31T22:29:53.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=8146' '!=' 'object_number=173, tag_tid=26, entry_tid=8053' 2024-08-31T22:29:53.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:53.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=8146' = 'object_number=173, tag_tid=26, entry_tid=8053' 2024-08-31T22:29:53.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=174, tag_tid=26, entry_tid=8146' 2024-08-31T22:29:53.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=8146' 2024-08-31T22:29:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=8146' 2024-08-31T22:29:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8146 2024-08-31T22:29:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 8146 2024-08-31T22:29:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=172, tag_tid=26, entry_tid=8236], [object_number=175, tag_tid=26, entry_tid=8235], [object_number=174, tag_tid=26, entry_tid=8234], [object_number=173, tag_tid=26, entry_tid=8233]]], state=connected] 2024-08-31T22:29:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=172, tag_tid=26, entry_tid=8236], [object_number=175, tag_tid=26, entry_tid=8235], [object_number=174, tag_tid=26, entry_tid=8234], [object_number=173, tag_tid=26, entry_tid=8233]]], state=connected] 2024-08-31T22:29:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=172, tag_tid=26, entry_tid=8236' 2024-08-31T22:29:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=172, tag_tid=26, entry_tid=8236' 2024-08-31T22:29:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=172, tag_tid=26, entry_tid=8236' '!=' 'object_number=174, tag_tid=26, entry_tid=8146' 2024-08-31T22:29:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=172, tag_tid=26, entry_tid=8236' = 'object_number=174, tag_tid=26, entry_tid=8146' 2024-08-31T22:29:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=172, tag_tid=26, entry_tid=8236' 2024-08-31T22:29:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:53.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:53.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=172, tag_tid=26, entry_tid=8236' 2024-08-31T22:29:53.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:53.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=172, tag_tid=26, entry_tid=8236' 2024-08-31T22:29:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8236 2024-08-31T22:29:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 8236 2024-08-31T22:29:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:53.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:53.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:53.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:53.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:54.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:54.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:54.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:54.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:54.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=8326], [object_number=173, tag_tid=26, entry_tid=8325], [object_number=172, tag_tid=26, entry_tid=8324], [object_number=175, tag_tid=26, entry_tid=8323]]], state=connected] 2024-08-31T22:29:54.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=8326], [object_number=173, tag_tid=26, entry_tid=8325], [object_number=172, tag_tid=26, entry_tid=8324], [object_number=175, tag_tid=26, entry_tid=8323]]], state=connected] 2024-08-31T22:29:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:54.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=174, tag_tid=26, entry_tid=8326' 2024-08-31T22:29:54.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=174, tag_tid=26, entry_tid=8326' 2024-08-31T22:29:54.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=8326' '!=' 'object_number=172, tag_tid=26, entry_tid=8236' 2024-08-31T22:29:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=8326' = 'object_number=172, tag_tid=26, entry_tid=8236' 2024-08-31T22:29:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=174, tag_tid=26, entry_tid=8326' 2024-08-31T22:29:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=8326' 2024-08-31T22:29:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=8326' 2024-08-31T22:29:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8326 2024-08-31T22:29:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 8326 2024-08-31T22:29:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:54.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:54.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:54.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:54.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:54.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:54.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:54.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:54.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:54.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=8466], [object_number=173, tag_tid=26, entry_tid=8465], [object_number=172, tag_tid=26, entry_tid=8464], [object_number=175, tag_tid=26, entry_tid=8463]]], state=connected] 2024-08-31T22:29:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=8466], [object_number=173, tag_tid=26, entry_tid=8465], [object_number=172, tag_tid=26, entry_tid=8464], [object_number=175, tag_tid=26, entry_tid=8463]]], state=connected] 2024-08-31T22:29:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=174, tag_tid=26, entry_tid=8466' 2024-08-31T22:29:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=174, tag_tid=26, entry_tid=8466' 2024-08-31T22:29:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=8466' '!=' 'object_number=174, tag_tid=26, entry_tid=8326' 2024-08-31T22:29:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=8466' = 'object_number=174, tag_tid=26, entry_tid=8326' 2024-08-31T22:29:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=174, tag_tid=26, entry_tid=8466' 2024-08-31T22:29:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=8466' 2024-08-31T22:29:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=8466' 2024-08-31T22:29:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8466 2024-08-31T22:29:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 8466 2024-08-31T22:29:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:54.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=173, tag_tid=26, entry_tid=8549], [object_number=172, tag_tid=26, entry_tid=8548], [object_number=175, tag_tid=26, entry_tid=8547], [object_number=174, tag_tid=26, entry_tid=8546]]], state=connected] 2024-08-31T22:29:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=173, tag_tid=26, entry_tid=8549], [object_number=172, tag_tid=26, entry_tid=8548], [object_number=175, tag_tid=26, entry_tid=8547], [object_number=174, tag_tid=26, entry_tid=8546]]], state=connected] 2024-08-31T22:29:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=173, tag_tid=26, entry_tid=8549' 2024-08-31T22:29:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=173, tag_tid=26, entry_tid=8549' 2024-08-31T22:29:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=173, tag_tid=26, entry_tid=8549' '!=' 'object_number=174, tag_tid=26, entry_tid=8466' 2024-08-31T22:29:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=173, tag_tid=26, entry_tid=8549' = 'object_number=174, tag_tid=26, entry_tid=8466' 2024-08-31T22:29:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=173, tag_tid=26, entry_tid=8549' 2024-08-31T22:29:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=173, tag_tid=26, entry_tid=8549' 2024-08-31T22:29:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=173, tag_tid=26, entry_tid=8549' 2024-08-31T22:29:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8549 2024-08-31T22:29:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 8549 2024-08-31T22:29:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=8639], [object_number=174, tag_tid=26, entry_tid=8638], [object_number=173, tag_tid=26, entry_tid=8637], [object_number=172, tag_tid=26, entry_tid=8636]]], state=connected] 2024-08-31T22:29:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:55.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:55.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=175, tag_tid=26, entry_tid=8639], [object_number=174, tag_tid=26, entry_tid=8638], [object_number=173, tag_tid=26, entry_tid=8637], [object_number=172, tag_tid=26, entry_tid=8636]]], state=connected] 2024-08-31T22:29:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=175, tag_tid=26, entry_tid=8639' 2024-08-31T22:29:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=175, tag_tid=26, entry_tid=8639' 2024-08-31T22:29:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=8639' '!=' 'object_number=173, tag_tid=26, entry_tid=8549' 2024-08-31T22:29:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=175, tag_tid=26, entry_tid=8639' = 'object_number=173, tag_tid=26, entry_tid=8549' 2024-08-31T22:29:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=175, tag_tid=26, entry_tid=8639' 2024-08-31T22:29:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:55.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=8639' 2024-08-31T22:29:55.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:55.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=175, tag_tid=26, entry_tid=8639' 2024-08-31T22:29:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8639 2024-08-31T22:29:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 8639 2024-08-31T22:29:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:56.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:56.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:56.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=173, tag_tid=26, entry_tid=8725], [object_number=172, tag_tid=26, entry_tid=8724], [object_number=175, tag_tid=26, entry_tid=8723], [object_number=174, tag_tid=26, entry_tid=8722]]], state=connected] 2024-08-31T22:29:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:56.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:56.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=173, tag_tid=26, entry_tid=8725], [object_number=172, tag_tid=26, entry_tid=8724], [object_number=175, tag_tid=26, entry_tid=8723], [object_number=174, tag_tid=26, entry_tid=8722]]], state=connected] 2024-08-31T22:29:56.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=173, tag_tid=26, entry_tid=8725' 2024-08-31T22:29:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=173, tag_tid=26, entry_tid=8725' 2024-08-31T22:29:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=173, tag_tid=26, entry_tid=8725' '!=' 'object_number=175, tag_tid=26, entry_tid=8639' 2024-08-31T22:29:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=173, tag_tid=26, entry_tid=8725' = 'object_number=175, tag_tid=26, entry_tid=8639' 2024-08-31T22:29:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=173, tag_tid=26, entry_tid=8725' 2024-08-31T22:29:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=173, tag_tid=26, entry_tid=8725' 2024-08-31T22:29:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=173, tag_tid=26, entry_tid=8725' 2024-08-31T22:29:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8725 2024-08-31T22:29:56.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 8725 2024-08-31T22:29:56.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:56.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:56.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=173, tag_tid=26, entry_tid=8809], [object_number=172, tag_tid=26, entry_tid=8808], [object_number=175, tag_tid=26, entry_tid=8807], [object_number=174, tag_tid=26, entry_tid=8806]]], state=connected] 2024-08-31T22:29:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=173, tag_tid=26, entry_tid=8809], [object_number=172, tag_tid=26, entry_tid=8808], [object_number=175, tag_tid=26, entry_tid=8807], [object_number=174, tag_tid=26, entry_tid=8806]]], state=connected] 2024-08-31T22:29:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=173, tag_tid=26, entry_tid=8809' 2024-08-31T22:29:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=173, tag_tid=26, entry_tid=8809' 2024-08-31T22:29:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=173, tag_tid=26, entry_tid=8809' '!=' 'object_number=173, tag_tid=26, entry_tid=8725' 2024-08-31T22:29:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=173, tag_tid=26, entry_tid=8809' = 'object_number=173, tag_tid=26, entry_tid=8725' 2024-08-31T22:29:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=173, tag_tid=26, entry_tid=8809' 2024-08-31T22:29:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=173, tag_tid=26, entry_tid=8809' 2024-08-31T22:29:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:56.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=173, tag_tid=26, entry_tid=8809' 2024-08-31T22:29:56.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:56.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8809 2024-08-31T22:29:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 8809 2024-08-31T22:29:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:56.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:56.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=8886], [object_number=173, tag_tid=26, entry_tid=8885], [object_number=172, tag_tid=26, entry_tid=8884], [object_number=175, tag_tid=26, entry_tid=8883]]], state=connected] 2024-08-31T22:29:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=8886], [object_number=173, tag_tid=26, entry_tid=8885], [object_number=172, tag_tid=26, entry_tid=8884], [object_number=175, tag_tid=26, entry_tid=8883]]], state=connected] 2024-08-31T22:29:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=174, tag_tid=26, entry_tid=8886' 2024-08-31T22:29:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=174, tag_tid=26, entry_tid=8886' 2024-08-31T22:29:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=8886' '!=' 'object_number=173, tag_tid=26, entry_tid=8809' 2024-08-31T22:29:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=8886' = 'object_number=173, tag_tid=26, entry_tid=8809' 2024-08-31T22:29:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=174, tag_tid=26, entry_tid=8886' 2024-08-31T22:29:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=8886' 2024-08-31T22:29:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=8886' 2024-08-31T22:29:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8886 2024-08-31T22:29:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 8886 2024-08-31T22:29:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:57.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:57.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:57.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=8946], [object_number=173, tag_tid=26, entry_tid=8945], [object_number=172, tag_tid=26, entry_tid=8944], [object_number=175, tag_tid=26, entry_tid=8943]]], state=connected] 2024-08-31T22:29:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:57.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:57.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:57.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:57.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:57.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:57.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=8946], [object_number=173, tag_tid=26, entry_tid=8945], [object_number=172, tag_tid=26, entry_tid=8944], [object_number=175, tag_tid=26, entry_tid=8943]]], state=connected] 2024-08-31T22:29:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=174, tag_tid=26, entry_tid=8946' 2024-08-31T22:29:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=174, tag_tid=26, entry_tid=8946' 2024-08-31T22:29:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=8946' '!=' 'object_number=174, tag_tid=26, entry_tid=8886' 2024-08-31T22:29:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=8946' = 'object_number=174, tag_tid=26, entry_tid=8886' 2024-08-31T22:29:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=174, tag_tid=26, entry_tid=8946' 2024-08-31T22:29:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=8946' 2024-08-31T22:29:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=8946' 2024-08-31T22:29:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8946 2024-08-31T22:29:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 8946 2024-08-31T22:29:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=172, tag_tid=26, entry_tid=9028], [object_number=175, tag_tid=26, entry_tid=9027], [object_number=174, tag_tid=26, entry_tid=9026], [object_number=173, tag_tid=26, entry_tid=9025]]], state=connected] 2024-08-31T22:29:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:58.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:58.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:58.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:58.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:58.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=172, tag_tid=26, entry_tid=9028], [object_number=175, tag_tid=26, entry_tid=9027], [object_number=174, tag_tid=26, entry_tid=9026], [object_number=173, tag_tid=26, entry_tid=9025]]], state=connected] 2024-08-31T22:29:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=172, tag_tid=26, entry_tid=9028' 2024-08-31T22:29:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=172, tag_tid=26, entry_tid=9028' 2024-08-31T22:29:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=172, tag_tid=26, entry_tid=9028' '!=' 'object_number=174, tag_tid=26, entry_tid=8946' 2024-08-31T22:29:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=172, tag_tid=26, entry_tid=9028' = 'object_number=174, tag_tid=26, entry_tid=8946' 2024-08-31T22:29:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=172, tag_tid=26, entry_tid=9028' 2024-08-31T22:29:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:58.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=172, tag_tid=26, entry_tid=9028' 2024-08-31T22:29:58.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:58.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:58.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:58.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=172, tag_tid=26, entry_tid=9028' 2024-08-31T22:29:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9028 2024-08-31T22:29:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 9028 2024-08-31T22:29:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=9074], [object_number=173, tag_tid=26, entry_tid=9073], [object_number=172, tag_tid=26, entry_tid=9072], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:58.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=174, tag_tid=26, entry_tid=9074], [object_number=173, tag_tid=26, entry_tid=9073], [object_number=172, tag_tid=26, entry_tid=9072], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=174, tag_tid=26, entry_tid=9074' 2024-08-31T22:29:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=174, tag_tid=26, entry_tid=9074' 2024-08-31T22:29:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=9074' '!=' 'object_number=172, tag_tid=26, entry_tid=9028' 2024-08-31T22:29:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:29:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=174, tag_tid=26, entry_tid=9074' = 'object_number=172, tag_tid=26, entry_tid=9028' 2024-08-31T22:29:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=174, tag_tid=26, entry_tid=9074' 2024-08-31T22:29:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=27 2024-08-31T22:29:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=9074' 2024-08-31T22:29:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:29:58.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:58.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=26 2024-08-31T22:29:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:29:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=174, tag_tid=26, entry_tid=9074' 2024-08-31T22:29:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:29:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:29:58.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9074 2024-08-31T22:29:58.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 27 = 26 -a 1 -le 9074 2024-08-31T22:29:58.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:29:58.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:29:58.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:29:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:29:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:29:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:29:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:29:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:29:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:29:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:29:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:29:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:29:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:29:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:29:58.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:29:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:29:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:29:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:29:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:29:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:29:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:29:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:29:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:29:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:29:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:29:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:29:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:29:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:29:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:29:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:29:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:29:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:29:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:29:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:29:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:29:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:29:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:29:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:29:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:29:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:29:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:29:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:29:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:29:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:29:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:29:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:29:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:29:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:29:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:29:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:29:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:29:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:29:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:29:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:29:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:29:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:29:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:29:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:29:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:29:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 43 2024-08-31T22:29:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 44 2024-08-31T22:29:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:29:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=177, tag_tid=27, entry_tid=1], [object_number=176, tag_tid=27, entry_tid=0], [object_number=174, tag_tid=26, entry_tid=9074], [object_number=175, tag_tid=26, entry_tid=9071]]], state=connected] 2024-08-31T22:29:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:29:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=177, tag_tid=27, entry_tid=1' -a 'object_number=177, tag_tid=27, entry_tid=1' = 'object_number=177, tag_tid=27, entry_tid=1' 2024-08-31T22:29:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:29:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-08-31T22:29:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:29:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:29:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:29:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap9 2024-08-31T22:29:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:29:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:29:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:30:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-08-31T22:30:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'test': 2024-08-31T22:30:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:30:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:30:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 17 2024-08-31T22:30:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a87b4c6a1b 2024-08-31T22:30:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a87b4c6a1b 2024-08-31T22:30:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:30:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:30:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:30:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:30:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:14:57 2024 2024-08-31T22:30:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:28:54 2024 2024-08-31T22:30:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:29:11 2024 2024-08-31T22:30:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:30:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a87b4c6a1b 2024-08-31T22:30:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:30:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:30:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:30:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:30:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n snap8-clean ']' 2024-08-31T22:30:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap8-clean 2024-08-31T22:30:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap8-clean 2024-08-31T22:30:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:30:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:30:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:30:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:30:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:30:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:30:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:30:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:30:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:30:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:30:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap9 2024-08-31T22:30:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap9 2024-08-31T22:30:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:30:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:30:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:30:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:30:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:30:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:30:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:30:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:30:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:30:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:30:05.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:05.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ clean_snap_name=snap9-clean 2024-08-31T22:30:05.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-08-31T22:30:05.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:30:05.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:05.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:05.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap9-clean 2024-08-31T22:30:05.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-08-31T22:30:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:30:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror test 2024-08-31T22:30:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:30:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:30:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=7 2024-08-31T22:30:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:30:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-08-31T22:30:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 42385408 bytes 2024-08-31T22:30:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 118929408 bytes 2024-08-31T22:30:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:30:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:30:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:30:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-08-31T22:30:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:30:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:30:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:30:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:30:14.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-08-31T22:30:14.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:14.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:14.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:14.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:30:14.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local description_pattern= 2024-08-31T22:30:14.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local service_pattern= 2024-08-31T22:30:14.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:30:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:30:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-08-31T22:30:14.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='test: 2024-08-31T22:30:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: global_id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:30:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+replaying 2024-08-31T22:30:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: replaying, {"bytes_per_second":2804463.07,"entries_behind_primary":526,"entries_per_second":172.03,"non_primary_position":{"entry_tid":8549,"object_number":173,"tag_tid":26},"primary_position":{"entry_tid":1,"object_number":177,"tag_tid":27},"seconds_until_synced":3} 2024-08-31T22:30:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: service: mirror.0 on smithi032 2024-08-31T22:30:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:29:55 2024-08-31T22:30:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: peer_sites: 2024-08-31T22:30:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: name: cluster2 2024-08-31T22:30:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: state: up+stopped 2024-08-31T22:30:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: description: local image is primary 2024-08-31T22:30:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: last_update: 2024-08-31 22:30:10' 2024-08-31T22:30:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ state: .*up+replaying' 2024-08-31T22:30:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: state: up+replaying 2024-08-31T22:30:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^ description: .*' 2024-08-31T22:30:14.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: description: replaying, {"bytes_per_second":2804463.07,"entries_behind_primary":526,"entries_per_second":172.03,"non_primary_position":{"entry_tid":8549,"object_number":173,"tag_tid":26},"primary_position":{"entry_tid":1,"object_number":177,"tag_tid":27},"seconds_until_synced":3} 2024-08-31T22:30:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:30:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:30:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep '^up+' 2024-08-31T22:30:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:30:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep 'service: *mirror..* on ' 2024-08-31T22:30:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: service: mirror.0 on smithi032 2024-08-31T22:30:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update 2024-08-31T22:30:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-08-31T22:30:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:29:55' 2024-08-31T22:30:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-08-31 22:29:55' 2024-08-31T22:30:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state_pattern=up+replaying 2024-08-31T22:30:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'prev_last_update=2024-08-31 22:29:55' 2024-08-31T22:30:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status 2024-08-31T22:30:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ CEPH_ARGS= 2024-08-31T22:30:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-08-31T22:30:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status='OKOKOK142574264mirror.0smithi03219.1.1-244-g3091dfc0trueOK42614267mirror.1smithi03219.1.1-244-g3091dfc0falseOKtestdd2c705e-25cf-4565-981f-d48de48eee84up+replayingreplaying, {"bytes_per_second":2804463.07,"entries_behind_primary":526,"entries_per_second":172.03,"non_primary_position":{"entry_tid":8549,"object_number":173,"tag_tid":26},"primary_position":{"entry_tid":1,"object_number":177,"tag_tid":27},"seconds_until_synced":3}42574264mirror.0smithi0322024-08-31 22:29:55cluster24ca32b00-2164-486e-837e-e93795a16af1up+stoppedlocal image is primary2024-08-31 22:30:10' 2024-08-31T22:30:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_update state 2024-08-31T22:30:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-08-31T22:30:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_update='2024-08-31 22:29:55' 2024-08-31T22:30:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-08-31T22:30:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ state=up+replaying 2024-08-31T22:30:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo up+replaying 2024-08-31T22:30:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep up+replaying 2024-08-31T22:30:14.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:up+replaying 2024-08-31T22:30:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap10 2024-08-31T22:30:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap10 2024-08-31T22:30:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:30:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap10 2024-08-31T22:30:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:30:45.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-08-31T22:30:45.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:45.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-08-31T22:30:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:30:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:30:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:30:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror test started 2024-08-31T22:30:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:30:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:30:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:30:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-08-31T22:30:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:30:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:30:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:30:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:30:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:30:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:30:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:30:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-08-31T22:30:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:30:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:30:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-08-31T22:30:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:30:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:30:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:30:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:30:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:30:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:30:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:30:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:30:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:30:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:30:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:30:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:30:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:30:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:30:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:30:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:30:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=182, tag_tid=29, entry_tid=5706], [object_number=181, tag_tid=29, entry_tid=5705], [object_number=180, tag_tid=29, entry_tid=5704], [object_number=183, tag_tid=29, entry_tid=5703]]], state=connected] 2024-08-31T22:30:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=182, tag_tid=29, entry_tid=5706], [object_number=181, tag_tid=29, entry_tid=5705], [object_number=180, tag_tid=29, entry_tid=5704], [object_number=183, tag_tid=29, entry_tid=5703]]], state=connected] 2024-08-31T22:30:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=182, tag_tid=29, entry_tid=5706' 2024-08-31T22:30:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=182, tag_tid=29, entry_tid=5706' 2024-08-31T22:30:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=182, tag_tid=29, entry_tid=5706' '!=' '' 2024-08-31T22:30:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=182, tag_tid=29, entry_tid=5706' = '' 2024-08-31T22:30:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=182, tag_tid=29, entry_tid=5706' 2024-08-31T22:30:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=182, tag_tid=29, entry_tid=5706' 2024-08-31T22:30:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=182, tag_tid=29, entry_tid=5706' 2024-08-31T22:30:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=5706 2024-08-31T22:30:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 5706 2024-08-31T22:30:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=181, tag_tid=29, entry_tid=5805], [object_number=180, tag_tid=29, entry_tid=5804], [object_number=183, tag_tid=29, entry_tid=5803], [object_number=182, tag_tid=29, entry_tid=5802]]], state=connected] 2024-08-31T22:30:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:47.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:47.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=181, tag_tid=29, entry_tid=5805], [object_number=180, tag_tid=29, entry_tid=5804], [object_number=183, tag_tid=29, entry_tid=5803], [object_number=182, tag_tid=29, entry_tid=5802]]], state=connected] 2024-08-31T22:30:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=181, tag_tid=29, entry_tid=5805' 2024-08-31T22:30:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=181, tag_tid=29, entry_tid=5805' 2024-08-31T22:30:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=181, tag_tid=29, entry_tid=5805' '!=' 'object_number=182, tag_tid=29, entry_tid=5706' 2024-08-31T22:30:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=181, tag_tid=29, entry_tid=5805' = 'object_number=182, tag_tid=29, entry_tid=5706' 2024-08-31T22:30:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=181, tag_tid=29, entry_tid=5805' 2024-08-31T22:30:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=181, tag_tid=29, entry_tid=5805' 2024-08-31T22:30:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=181, tag_tid=29, entry_tid=5805' 2024-08-31T22:30:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:47.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=5805 2024-08-31T22:30:47.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 5805 2024-08-31T22:30:47.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:47.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=183, tag_tid=29, entry_tid=5871], [object_number=182, tag_tid=29, entry_tid=5870], [object_number=181, tag_tid=29, entry_tid=5869], [object_number=180, tag_tid=29, entry_tid=5868]]], state=connected] 2024-08-31T22:30:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=183, tag_tid=29, entry_tid=5871], [object_number=182, tag_tid=29, entry_tid=5870], [object_number=181, tag_tid=29, entry_tid=5869], [object_number=180, tag_tid=29, entry_tid=5868]]], state=connected] 2024-08-31T22:30:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=183, tag_tid=29, entry_tid=5871' 2024-08-31T22:30:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=183, tag_tid=29, entry_tid=5871' 2024-08-31T22:30:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=183, tag_tid=29, entry_tid=5871' '!=' 'object_number=181, tag_tid=29, entry_tid=5805' 2024-08-31T22:30:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=183, tag_tid=29, entry_tid=5871' = 'object_number=181, tag_tid=29, entry_tid=5805' 2024-08-31T22:30:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=183, tag_tid=29, entry_tid=5871' 2024-08-31T22:30:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:47.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=183, tag_tid=29, entry_tid=5871' 2024-08-31T22:30:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=183, tag_tid=29, entry_tid=5871' 2024-08-31T22:30:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:47.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=5871 2024-08-31T22:30:47.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 5871 2024-08-31T22:30:47.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:47.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:47.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=180, tag_tid=29, entry_tid=5988], [object_number=183, tag_tid=29, entry_tid=5987], [object_number=182, tag_tid=29, entry_tid=5986], [object_number=181, tag_tid=29, entry_tid=5985]]], state=connected] 2024-08-31T22:30:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=180, tag_tid=29, entry_tid=5988], [object_number=183, tag_tid=29, entry_tid=5987], [object_number=182, tag_tid=29, entry_tid=5986], [object_number=181, tag_tid=29, entry_tid=5985]]], state=connected] 2024-08-31T22:30:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=180, tag_tid=29, entry_tid=5988' 2024-08-31T22:30:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=180, tag_tid=29, entry_tid=5988' 2024-08-31T22:30:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=180, tag_tid=29, entry_tid=5988' '!=' 'object_number=183, tag_tid=29, entry_tid=5871' 2024-08-31T22:30:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=180, tag_tid=29, entry_tid=5988' = 'object_number=183, tag_tid=29, entry_tid=5871' 2024-08-31T22:30:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=180, tag_tid=29, entry_tid=5988' 2024-08-31T22:30:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=180, tag_tid=29, entry_tid=5988' 2024-08-31T22:30:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=180, tag_tid=29, entry_tid=5988' 2024-08-31T22:30:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=5988 2024-08-31T22:30:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 5988 2024-08-31T22:30:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=183, tag_tid=29, entry_tid=6075], [object_number=182, tag_tid=29, entry_tid=6074], [object_number=181, tag_tid=29, entry_tid=6073], [object_number=180, tag_tid=29, entry_tid=6072]]], state=connected] 2024-08-31T22:30:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=183, tag_tid=29, entry_tid=6075], [object_number=182, tag_tid=29, entry_tid=6074], [object_number=181, tag_tid=29, entry_tid=6073], [object_number=180, tag_tid=29, entry_tid=6072]]], state=connected] 2024-08-31T22:30:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=183, tag_tid=29, entry_tid=6075' 2024-08-31T22:30:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=183, tag_tid=29, entry_tid=6075' 2024-08-31T22:30:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=183, tag_tid=29, entry_tid=6075' '!=' 'object_number=180, tag_tid=29, entry_tid=5988' 2024-08-31T22:30:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=183, tag_tid=29, entry_tid=6075' = 'object_number=180, tag_tid=29, entry_tid=5988' 2024-08-31T22:30:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=183, tag_tid=29, entry_tid=6075' 2024-08-31T22:30:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=183, tag_tid=29, entry_tid=6075' 2024-08-31T22:30:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=183, tag_tid=29, entry_tid=6075' 2024-08-31T22:30:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6075 2024-08-31T22:30:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 6075 2024-08-31T22:30:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=181, tag_tid=29, entry_tid=6157], [object_number=180, tag_tid=29, entry_tid=6156], [object_number=183, tag_tid=29, entry_tid=6155], [object_number=182, tag_tid=29, entry_tid=6154]]], state=connected] 2024-08-31T22:30:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=181, tag_tid=29, entry_tid=6157], [object_number=180, tag_tid=29, entry_tid=6156], [object_number=183, tag_tid=29, entry_tid=6155], [object_number=182, tag_tid=29, entry_tid=6154]]], state=connected] 2024-08-31T22:30:49.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=181, tag_tid=29, entry_tid=6157' 2024-08-31T22:30:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=181, tag_tid=29, entry_tid=6157' 2024-08-31T22:30:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=181, tag_tid=29, entry_tid=6157' '!=' 'object_number=183, tag_tid=29, entry_tid=6075' 2024-08-31T22:30:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=181, tag_tid=29, entry_tid=6157' = 'object_number=183, tag_tid=29, entry_tid=6075' 2024-08-31T22:30:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=181, tag_tid=29, entry_tid=6157' 2024-08-31T22:30:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=181, tag_tid=29, entry_tid=6157' 2024-08-31T22:30:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:49.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=181, tag_tid=29, entry_tid=6157' 2024-08-31T22:30:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:49.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6157 2024-08-31T22:30:49.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 6157 2024-08-31T22:30:49.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:49.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:49.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:49.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:49.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:49.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:49.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:49.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:49.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=183, tag_tid=29, entry_tid=6247], [object_number=182, tag_tid=29, entry_tid=6246], [object_number=181, tag_tid=29, entry_tid=6245], [object_number=180, tag_tid=29, entry_tid=6244]]], state=connected] 2024-08-31T22:30:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=183, tag_tid=29, entry_tid=6247], [object_number=182, tag_tid=29, entry_tid=6246], [object_number=181, tag_tid=29, entry_tid=6245], [object_number=180, tag_tid=29, entry_tid=6244]]], state=connected] 2024-08-31T22:30:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=183, tag_tid=29, entry_tid=6247' 2024-08-31T22:30:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=183, tag_tid=29, entry_tid=6247' 2024-08-31T22:30:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=183, tag_tid=29, entry_tid=6247' '!=' 'object_number=181, tag_tid=29, entry_tid=6157' 2024-08-31T22:30:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=183, tag_tid=29, entry_tid=6247' = 'object_number=181, tag_tid=29, entry_tid=6157' 2024-08-31T22:30:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=183, tag_tid=29, entry_tid=6247' 2024-08-31T22:30:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=183, tag_tid=29, entry_tid=6247' 2024-08-31T22:30:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=183, tag_tid=29, entry_tid=6247' 2024-08-31T22:30:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6247 2024-08-31T22:30:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 6247 2024-08-31T22:30:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:50.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:50.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=183, tag_tid=29, entry_tid=6335], [object_number=182, tag_tid=29, entry_tid=6334], [object_number=181, tag_tid=29, entry_tid=6333], [object_number=180, tag_tid=29, entry_tid=6332]]], state=connected] 2024-08-31T22:30:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=183, tag_tid=29, entry_tid=6335], [object_number=182, tag_tid=29, entry_tid=6334], [object_number=181, tag_tid=29, entry_tid=6333], [object_number=180, tag_tid=29, entry_tid=6332]]], state=connected] 2024-08-31T22:30:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=183, tag_tid=29, entry_tid=6335' 2024-08-31T22:30:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=183, tag_tid=29, entry_tid=6335' 2024-08-31T22:30:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=183, tag_tid=29, entry_tid=6335' '!=' 'object_number=183, tag_tid=29, entry_tid=6247' 2024-08-31T22:30:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=183, tag_tid=29, entry_tid=6335' = 'object_number=183, tag_tid=29, entry_tid=6247' 2024-08-31T22:30:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=183, tag_tid=29, entry_tid=6335' 2024-08-31T22:30:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=183, tag_tid=29, entry_tid=6335' 2024-08-31T22:30:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:50.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:50.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=183, tag_tid=29, entry_tid=6335' 2024-08-31T22:30:50.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:50.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:50.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6335 2024-08-31T22:30:50.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 6335 2024-08-31T22:30:50.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:50.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:50.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=180, tag_tid=29, entry_tid=6420], [object_number=183, tag_tid=29, entry_tid=6419], [object_number=182, tag_tid=29, entry_tid=6418], [object_number=181, tag_tid=29, entry_tid=6417]]], state=connected] 2024-08-31T22:30:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=180, tag_tid=29, entry_tid=6420], [object_number=183, tag_tid=29, entry_tid=6419], [object_number=182, tag_tid=29, entry_tid=6418], [object_number=181, tag_tid=29, entry_tid=6417]]], state=connected] 2024-08-31T22:30:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=180, tag_tid=29, entry_tid=6420' 2024-08-31T22:30:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=180, tag_tid=29, entry_tid=6420' 2024-08-31T22:30:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=180, tag_tid=29, entry_tid=6420' '!=' 'object_number=183, tag_tid=29, entry_tid=6335' 2024-08-31T22:30:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=180, tag_tid=29, entry_tid=6420' = 'object_number=183, tag_tid=29, entry_tid=6335' 2024-08-31T22:30:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=180, tag_tid=29, entry_tid=6420' 2024-08-31T22:30:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=180, tag_tid=29, entry_tid=6420' 2024-08-31T22:30:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=180, tag_tid=29, entry_tid=6420' 2024-08-31T22:30:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6420 2024-08-31T22:30:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 6420 2024-08-31T22:30:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=180, tag_tid=29, entry_tid=6504], [object_number=183, tag_tid=29, entry_tid=6503], [object_number=182, tag_tid=29, entry_tid=6502], [object_number=181, tag_tid=29, entry_tid=6501]]], state=connected] 2024-08-31T22:30:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=180, tag_tid=29, entry_tid=6504], [object_number=183, tag_tid=29, entry_tid=6503], [object_number=182, tag_tid=29, entry_tid=6502], [object_number=181, tag_tid=29, entry_tid=6501]]], state=connected] 2024-08-31T22:30:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=180, tag_tid=29, entry_tid=6504' 2024-08-31T22:30:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=180, tag_tid=29, entry_tid=6504' 2024-08-31T22:30:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=180, tag_tid=29, entry_tid=6504' '!=' 'object_number=180, tag_tid=29, entry_tid=6420' 2024-08-31T22:30:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=180, tag_tid=29, entry_tid=6504' = 'object_number=180, tag_tid=29, entry_tid=6420' 2024-08-31T22:30:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=180, tag_tid=29, entry_tid=6504' 2024-08-31T22:30:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=180, tag_tid=29, entry_tid=6504' 2024-08-31T22:30:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:51.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:51.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:51.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:51.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=180, tag_tid=29, entry_tid=6504' 2024-08-31T22:30:51.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:51.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6504 2024-08-31T22:30:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 6504 2024-08-31T22:30:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:51.668 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:30:51.668+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:30:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:51.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:51.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:51.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:51.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:51.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:51.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:51.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=180, tag_tid=29, entry_tid=6596], [object_number=183, tag_tid=29, entry_tid=6595], [object_number=182, tag_tid=29, entry_tid=6594], [object_number=181, tag_tid=29, entry_tid=6593]]], state=connected] 2024-08-31T22:30:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=180, tag_tid=29, entry_tid=6596], [object_number=183, tag_tid=29, entry_tid=6595], [object_number=182, tag_tid=29, entry_tid=6594], [object_number=181, tag_tid=29, entry_tid=6593]]], state=connected] 2024-08-31T22:30:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=180, tag_tid=29, entry_tid=6596' 2024-08-31T22:30:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=180, tag_tid=29, entry_tid=6596' 2024-08-31T22:30:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=180, tag_tid=29, entry_tid=6596' '!=' 'object_number=180, tag_tid=29, entry_tid=6504' 2024-08-31T22:30:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=180, tag_tid=29, entry_tid=6596' = 'object_number=180, tag_tid=29, entry_tid=6504' 2024-08-31T22:30:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=180, tag_tid=29, entry_tid=6596' 2024-08-31T22:30:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=180, tag_tid=29, entry_tid=6596' 2024-08-31T22:30:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=180, tag_tid=29, entry_tid=6596' 2024-08-31T22:30:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6596 2024-08-31T22:30:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 6596 2024-08-31T22:30:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:52.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=186, tag_tid=29, entry_tid=6674], [object_number=185, tag_tid=29, entry_tid=6673], [object_number=184, tag_tid=29, entry_tid=6672], [object_number=183, tag_tid=29, entry_tid=6671]]], state=connected] 2024-08-31T22:30:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:52.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:52.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:52.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 45 2024-08-31T22:30:52.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:52.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:52.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:52.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=186, tag_tid=29, entry_tid=6674], [object_number=185, tag_tid=29, entry_tid=6673], [object_number=184, tag_tid=29, entry_tid=6672], [object_number=183, tag_tid=29, entry_tid=6671]]], state=connected] 2024-08-31T22:30:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=186, tag_tid=29, entry_tid=6674' 2024-08-31T22:30:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=186, tag_tid=29, entry_tid=6674' 2024-08-31T22:30:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=186, tag_tid=29, entry_tid=6674' '!=' 'object_number=180, tag_tid=29, entry_tid=6596' 2024-08-31T22:30:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=186, tag_tid=29, entry_tid=6674' = 'object_number=180, tag_tid=29, entry_tid=6596' 2024-08-31T22:30:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=186, tag_tid=29, entry_tid=6674' 2024-08-31T22:30:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=186, tag_tid=29, entry_tid=6674' 2024-08-31T22:30:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=186, tag_tid=29, entry_tid=6674' 2024-08-31T22:30:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6674 2024-08-31T22:30:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 6674 2024-08-31T22:30:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:52.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=187, tag_tid=29, entry_tid=6763], [object_number=186, tag_tid=29, entry_tid=6762], [object_number=185, tag_tid=29, entry_tid=6761], [object_number=184, tag_tid=29, entry_tid=6760]]], state=connected] 2024-08-31T22:30:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=187, tag_tid=29, entry_tid=6763], [object_number=186, tag_tid=29, entry_tid=6762], [object_number=185, tag_tid=29, entry_tid=6761], [object_number=184, tag_tid=29, entry_tid=6760]]], state=connected] 2024-08-31T22:30:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=187, tag_tid=29, entry_tid=6763' 2024-08-31T22:30:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=187, tag_tid=29, entry_tid=6763' 2024-08-31T22:30:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=187, tag_tid=29, entry_tid=6763' '!=' 'object_number=186, tag_tid=29, entry_tid=6674' 2024-08-31T22:30:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=187, tag_tid=29, entry_tid=6763' = 'object_number=186, tag_tid=29, entry_tid=6674' 2024-08-31T22:30:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=187, tag_tid=29, entry_tid=6763' 2024-08-31T22:30:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=187, tag_tid=29, entry_tid=6763' 2024-08-31T22:30:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:53.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:53.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:53.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:53.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=187, tag_tid=29, entry_tid=6763' 2024-08-31T22:30:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6763 2024-08-31T22:30:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 6763 2024-08-31T22:30:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:53.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:53.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:53.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=185, tag_tid=29, entry_tid=6853], [object_number=184, tag_tid=29, entry_tid=6852], [object_number=187, tag_tid=29, entry_tid=6851], [object_number=186, tag_tid=29, entry_tid=6850]]], state=connected] 2024-08-31T22:30:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=185, tag_tid=29, entry_tid=6853], [object_number=184, tag_tid=29, entry_tid=6852], [object_number=187, tag_tid=29, entry_tid=6851], [object_number=186, tag_tid=29, entry_tid=6850]]], state=connected] 2024-08-31T22:30:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=185, tag_tid=29, entry_tid=6853' 2024-08-31T22:30:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=185, tag_tid=29, entry_tid=6853' 2024-08-31T22:30:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=185, tag_tid=29, entry_tid=6853' '!=' 'object_number=187, tag_tid=29, entry_tid=6763' 2024-08-31T22:30:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=185, tag_tid=29, entry_tid=6853' = 'object_number=187, tag_tid=29, entry_tid=6763' 2024-08-31T22:30:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=185, tag_tid=29, entry_tid=6853' 2024-08-31T22:30:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=29, entry_tid=6853' 2024-08-31T22:30:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=29, entry_tid=6853' 2024-08-31T22:30:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6853 2024-08-31T22:30:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 6853 2024-08-31T22:30:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:53.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=185, tag_tid=29, entry_tid=6941], [object_number=184, tag_tid=29, entry_tid=6940], [object_number=187, tag_tid=29, entry_tid=6939], [object_number=186, tag_tid=29, entry_tid=6938]]], state=connected] 2024-08-31T22:30:53.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=185, tag_tid=29, entry_tid=6941], [object_number=184, tag_tid=29, entry_tid=6940], [object_number=187, tag_tid=29, entry_tid=6939], [object_number=186, tag_tid=29, entry_tid=6938]]], state=connected] 2024-08-31T22:30:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=185, tag_tid=29, entry_tid=6941' 2024-08-31T22:30:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=185, tag_tid=29, entry_tid=6941' 2024-08-31T22:30:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=185, tag_tid=29, entry_tid=6941' '!=' 'object_number=185, tag_tid=29, entry_tid=6853' 2024-08-31T22:30:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=185, tag_tid=29, entry_tid=6941' = 'object_number=185, tag_tid=29, entry_tid=6853' 2024-08-31T22:30:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=185, tag_tid=29, entry_tid=6941' 2024-08-31T22:30:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=29, entry_tid=6941' 2024-08-31T22:30:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=29, entry_tid=6941' 2024-08-31T22:30:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=6941 2024-08-31T22:30:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 6941 2024-08-31T22:30:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:54.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:54.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:54.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:54.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:54.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:54.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=186, tag_tid=29, entry_tid=7046], [object_number=185, tag_tid=29, entry_tid=7045], [object_number=184, tag_tid=29, entry_tid=7044], [object_number=187, tag_tid=29, entry_tid=7043]]], state=connected] 2024-08-31T22:30:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=186, tag_tid=29, entry_tid=7046], [object_number=185, tag_tid=29, entry_tid=7045], [object_number=184, tag_tid=29, entry_tid=7044], [object_number=187, tag_tid=29, entry_tid=7043]]], state=connected] 2024-08-31T22:30:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:54.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=186, tag_tid=29, entry_tid=7046' 2024-08-31T22:30:54.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=186, tag_tid=29, entry_tid=7046' 2024-08-31T22:30:54.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=186, tag_tid=29, entry_tid=7046' '!=' 'object_number=185, tag_tid=29, entry_tid=6941' 2024-08-31T22:30:54.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:54.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=186, tag_tid=29, entry_tid=7046' = 'object_number=185, tag_tid=29, entry_tid=6941' 2024-08-31T22:30:54.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=186, tag_tid=29, entry_tid=7046' 2024-08-31T22:30:54.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:54.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:54.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:54.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:54.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=186, tag_tid=29, entry_tid=7046' 2024-08-31T22:30:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=186, tag_tid=29, entry_tid=7046' 2024-08-31T22:30:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7046 2024-08-31T22:30:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 7046 2024-08-31T22:30:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:54.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=185, tag_tid=29, entry_tid=7125], [object_number=184, tag_tid=29, entry_tid=7124], [object_number=187, tag_tid=29, entry_tid=7123], [object_number=186, tag_tid=29, entry_tid=7122]]], state=connected] 2024-08-31T22:30:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=185, tag_tid=29, entry_tid=7125], [object_number=184, tag_tid=29, entry_tid=7124], [object_number=187, tag_tid=29, entry_tid=7123], [object_number=186, tag_tid=29, entry_tid=7122]]], state=connected] 2024-08-31T22:30:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=185, tag_tid=29, entry_tid=7125' 2024-08-31T22:30:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=185, tag_tid=29, entry_tid=7125' 2024-08-31T22:30:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=185, tag_tid=29, entry_tid=7125' '!=' 'object_number=186, tag_tid=29, entry_tid=7046' 2024-08-31T22:30:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=185, tag_tid=29, entry_tid=7125' = 'object_number=186, tag_tid=29, entry_tid=7046' 2024-08-31T22:30:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=185, tag_tid=29, entry_tid=7125' 2024-08-31T22:30:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=29, entry_tid=7125' 2024-08-31T22:30:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=29, entry_tid=7125' 2024-08-31T22:30:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7125 2024-08-31T22:30:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 7125 2024-08-31T22:30:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=187, tag_tid=29, entry_tid=7219], [object_number=186, tag_tid=29, entry_tid=7218], [object_number=185, tag_tid=29, entry_tid=7217], [object_number=184, tag_tid=29, entry_tid=7216]]], state=connected] 2024-08-31T22:30:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:55.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:55.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:55.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:55.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:55.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=187, tag_tid=29, entry_tid=7219], [object_number=186, tag_tid=29, entry_tid=7218], [object_number=185, tag_tid=29, entry_tid=7217], [object_number=184, tag_tid=29, entry_tid=7216]]], state=connected] 2024-08-31T22:30:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=187, tag_tid=29, entry_tid=7219' 2024-08-31T22:30:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=187, tag_tid=29, entry_tid=7219' 2024-08-31T22:30:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=187, tag_tid=29, entry_tid=7219' '!=' 'object_number=185, tag_tid=29, entry_tid=7125' 2024-08-31T22:30:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=187, tag_tid=29, entry_tid=7219' = 'object_number=185, tag_tid=29, entry_tid=7125' 2024-08-31T22:30:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=187, tag_tid=29, entry_tid=7219' 2024-08-31T22:30:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=187, tag_tid=29, entry_tid=7219' 2024-08-31T22:30:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:55.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=187, tag_tid=29, entry_tid=7219' 2024-08-31T22:30:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7219 2024-08-31T22:30:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 7219 2024-08-31T22:30:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=184, tag_tid=29, entry_tid=7312], [object_number=187, tag_tid=29, entry_tid=7311], [object_number=186, tag_tid=29, entry_tid=7310], [object_number=185, tag_tid=29, entry_tid=7309]]], state=connected] 2024-08-31T22:30:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:55.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:55.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:55.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:55.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:55.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:55.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=184, tag_tid=29, entry_tid=7312], [object_number=187, tag_tid=29, entry_tid=7311], [object_number=186, tag_tid=29, entry_tid=7310], [object_number=185, tag_tid=29, entry_tid=7309]]], state=connected] 2024-08-31T22:30:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=184, tag_tid=29, entry_tid=7312' 2024-08-31T22:30:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=184, tag_tid=29, entry_tid=7312' 2024-08-31T22:30:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=184, tag_tid=29, entry_tid=7312' '!=' 'object_number=187, tag_tid=29, entry_tid=7219' 2024-08-31T22:30:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=184, tag_tid=29, entry_tid=7312' = 'object_number=187, tag_tid=29, entry_tid=7219' 2024-08-31T22:30:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=184, tag_tid=29, entry_tid=7312' 2024-08-31T22:30:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=184, tag_tid=29, entry_tid=7312' 2024-08-31T22:30:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=184, tag_tid=29, entry_tid=7312' 2024-08-31T22:30:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7312 2024-08-31T22:30:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 7312 2024-08-31T22:30:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=186, tag_tid=29, entry_tid=7398], [object_number=185, tag_tid=29, entry_tid=7397], [object_number=184, tag_tid=29, entry_tid=7396], [object_number=187, tag_tid=29, entry_tid=7395]]], state=connected] 2024-08-31T22:30:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=186, tag_tid=29, entry_tid=7398], [object_number=185, tag_tid=29, entry_tid=7397], [object_number=184, tag_tid=29, entry_tid=7396], [object_number=187, tag_tid=29, entry_tid=7395]]], state=connected] 2024-08-31T22:30:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=186, tag_tid=29, entry_tid=7398' 2024-08-31T22:30:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=186, tag_tid=29, entry_tid=7398' 2024-08-31T22:30:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=186, tag_tid=29, entry_tid=7398' '!=' 'object_number=184, tag_tid=29, entry_tid=7312' 2024-08-31T22:30:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=186, tag_tid=29, entry_tid=7398' = 'object_number=184, tag_tid=29, entry_tid=7312' 2024-08-31T22:30:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=186, tag_tid=29, entry_tid=7398' 2024-08-31T22:30:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=186, tag_tid=29, entry_tid=7398' 2024-08-31T22:30:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=186, tag_tid=29, entry_tid=7398' 2024-08-31T22:30:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7398 2024-08-31T22:30:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 7398 2024-08-31T22:30:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:56.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:56.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=186, tag_tid=29, entry_tid=7482], [object_number=185, tag_tid=29, entry_tid=7481], [object_number=184, tag_tid=29, entry_tid=7480], [object_number=187, tag_tid=29, entry_tid=7479]]], state=connected] 2024-08-31T22:30:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=186, tag_tid=29, entry_tid=7482], [object_number=185, tag_tid=29, entry_tid=7481], [object_number=184, tag_tid=29, entry_tid=7480], [object_number=187, tag_tid=29, entry_tid=7479]]], state=connected] 2024-08-31T22:30:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=186, tag_tid=29, entry_tid=7482' 2024-08-31T22:30:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=186, tag_tid=29, entry_tid=7482' 2024-08-31T22:30:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=186, tag_tid=29, entry_tid=7482' '!=' 'object_number=186, tag_tid=29, entry_tid=7398' 2024-08-31T22:30:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=186, tag_tid=29, entry_tid=7482' = 'object_number=186, tag_tid=29, entry_tid=7398' 2024-08-31T22:30:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=186, tag_tid=29, entry_tid=7482' 2024-08-31T22:30:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=186, tag_tid=29, entry_tid=7482' 2024-08-31T22:30:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=186, tag_tid=29, entry_tid=7482' 2024-08-31T22:30:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7482 2024-08-31T22:30:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 7482 2024-08-31T22:30:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:57.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:57.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:57.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=184, tag_tid=29, entry_tid=7536], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534], [object_number=185, tag_tid=29, entry_tid=7533]]], state=connected] 2024-08-31T22:30:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=184, tag_tid=29, entry_tid=7536], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534], [object_number=185, tag_tid=29, entry_tid=7533]]], state=connected] 2024-08-31T22:30:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=184, tag_tid=29, entry_tid=7536' 2024-08-31T22:30:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=184, tag_tid=29, entry_tid=7536' 2024-08-31T22:30:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=184, tag_tid=29, entry_tid=7536' '!=' 'object_number=186, tag_tid=29, entry_tid=7482' 2024-08-31T22:30:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:30:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=184, tag_tid=29, entry_tid=7536' = 'object_number=186, tag_tid=29, entry_tid=7482' 2024-08-31T22:30:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=184, tag_tid=29, entry_tid=7536' 2024-08-31T22:30:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=30 2024-08-31T22:30:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=184, tag_tid=29, entry_tid=7536' 2024-08-31T22:30:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:30:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=29 2024-08-31T22:30:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:57.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:30:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=184, tag_tid=29, entry_tid=7536' 2024-08-31T22:30:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:30:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:30:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7536 2024-08-31T22:30:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 30 = 29 -a 1 -le 7536 2024-08-31T22:30:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:30:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:30:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=184, tag_tid=29, entry_tid=7536], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534], [object_number=185, tag_tid=29, entry_tid=7533]]], state=connected] 2024-08-31T22:30:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=184, tag_tid=29, entry_tid=7536], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534], [object_number=185, tag_tid=29, entry_tid=7533]]], state=connected] 2024-08-31T22:30:58.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=184, tag_tid=29, entry_tid=7536' 2024-08-31T22:30:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=184, tag_tid=29, entry_tid=7536' 2024-08-31T22:30:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=184, tag_tid=29, entry_tid=7536' '!=' 'object_number=184, tag_tid=29, entry_tid=7536' 2024-08-31T22:30:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:30:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.4 2024-08-31T22:30:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:30:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:30:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:30:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:30:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:30:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:30:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:30:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:30:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:30:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:30:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:30:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:30:58.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:30:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:30:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:30:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:30:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:30:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:30:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:30:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:30:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:30:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:30:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:30:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:30:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:30:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:30:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:30:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:30:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:30:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:30:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:30:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:30:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:30:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:30:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:30:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:30:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:30:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:30:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:30:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:30:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:30:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:30:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:30:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:30:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:30:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:30:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:30:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:30:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:30:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:30:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:30:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:30:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:30:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:30:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:30:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:30:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:30:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:30:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:30:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:30:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:30:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=30, entry_tid=1' -a 'object_number=185, tag_tid=30, entry_tid=1' = 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:30:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-08-31T22:30:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:30:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap10 2024-08-31T22:30:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:30:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:30:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:30:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-08-31T22:30:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'test': 2024-08-31T22:30:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:30:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:30:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 19 2024-08-31T22:30:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a87b4c6a1b 2024-08-31T22:30:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a87b4c6a1b 2024-08-31T22:30:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:30:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:30:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:30:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:30:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:14:57 2024 2024-08-31T22:30:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:30:01 2024 2024-08-31T22:30:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:30:12 2024 2024-08-31T22:30:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:30:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a87b4c6a1b 2024-08-31T22:30:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:30:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:30:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:30:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:30:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:30:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n snap9-clean ']' 2024-08-31T22:30:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap9-clean 2024-08-31T22:30:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:30:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:30:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap9-clean 2024-08-31T22:30:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:30:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:30:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:30:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:30:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap10 2024-08-31T22:31:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap10 2024-08-31T22:31:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:02.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ clean_snap_name=snap10-clean 2024-08-31T22:31:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-08-31T22:31:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:31:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap10-clean 2024-08-31T22:31:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-08-31T22:31:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-08-31T22:31:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:31:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-08-31T22:31:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:31:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:31:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:31:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:31:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:31:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror test started 2024-08-31T22:31:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:31:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:31:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:31:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:31:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-08-31T22:31:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:31:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:31:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:31:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:31:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:31:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:31:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:31:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:31:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:31:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:31:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:31:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-08-31T22:31:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:31:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:31:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:31:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:31:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:31:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:31:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:31:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:31:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:31:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:31:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:31:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:31:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:31:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:31:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:31:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:31:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:31:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:31:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:31:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:31:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:31:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:31:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:31:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:31:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:31:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:31:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:31:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:31:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:31:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:31:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:31:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:31:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:31:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-08-31T22:31:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:31:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:31:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:31:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:31:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:31:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:31:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:31:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:31:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:31:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:31:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:31:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-08-31T22:31:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:31:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:31:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:31:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:31:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:31:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:31:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:31:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:31:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:31:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:31:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:31:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:31:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:31:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:31:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:31:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:31:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:31:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:31:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:31:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:31:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:31:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:31:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:31:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:31:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:31:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:31:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:31:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:31:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:31:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:31:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:31:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:31:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:31:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:31:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:31:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:31:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:31:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:31:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:31:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:31:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:31:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:31:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:31:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:31:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:31:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:31:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:31:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:31:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:31:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:31:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:31:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:31:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:31:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:31:06.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:31:06.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:31:06.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:31:06.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:31:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:31:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:31:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:31:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:31:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:31:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:31:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:31:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:31:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:31:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:31:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:31:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:31:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:31:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:31:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:31:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:31:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:31:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:31:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=31, entry_tid=1], [object_number=184, tag_tid=31, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:31:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:31:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:31:06.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=31, entry_tid=1' 2024-08-31T22:31:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:31:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:31:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:31:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:31:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:31:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:31:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:31:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:31:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:31:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:31:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:31:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:31:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:31:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:31:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:31:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:31:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:31:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:31:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=31, entry_tid=1], [object_number=184, tag_tid=31, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:31:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=185, tag_tid=30, entry_tid=1], [object_number=184, tag_tid=30, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:31:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:31:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:31:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=31, entry_tid=1' -a 'object_number=185, tag_tid=31, entry_tid=1' = 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:31:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=185, tag_tid=30, entry_tid=1' '!=' '' 2024-08-31T22:31:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:31:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=185, tag_tid=30, entry_tid=1' = '' 2024-08-31T22:31:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:31:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=31, entry_tid=1' 2024-08-31T22:31:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:31:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:31:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=31 2024-08-31T22:31:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:31:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:31:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:31:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=30 2024-08-31T22:31:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=31, entry_tid=1' 2024-08-31T22:31:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:31:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:31:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:31:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=185, tag_tid=30, entry_tid=1' 2024-08-31T22:31:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:31:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:31:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=1 2024-08-31T22:31:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 31 = 30 -a 1 -le 1 2024-08-31T22:31:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:31:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:31:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:31:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror test 2024-08-31T22:31:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:31:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:31:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n test ']' 2024-08-31T22:31:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/test' 2024-08-31T22:31:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-08-31T22:31:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:31:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:31:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:31:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:31:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:31:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:31:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:31:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:31:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:31:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:31:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-08-31T22:31:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:31:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:31:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:31:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:31:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:31:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:31:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:31:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:31:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:31:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:31:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:31:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:31:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:31:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:31:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:31:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:31:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:31:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:31:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:31:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:31:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:31:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:31:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:31:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:31:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:31:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:31:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:31:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:31:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:31:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:31:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:31:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:31:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:31:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-08-31T22:31:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror test 2024-08-31T22:31:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:31:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:31:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:31:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '' 2024-08-31T22:31:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:31:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:31:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:31:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:31:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:31:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:31:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:31:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:31:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:31:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:31:07.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:31:07.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:31:07.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:31:07.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=31, entry_tid=1], [object_number=184, tag_tid=31, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:31:07.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=185, tag_tid=31, entry_tid=1], [object_number=184, tag_tid=31, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:31:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:31:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=185, tag_tid=31, entry_tid=1' 2024-08-31T22:31:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-08-31T22:31:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:31:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:31:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:31:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-08-31T22:31:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:31:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:31:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:31:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:31:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-test.status 2024-08-31T22:31:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-test.status 2024-08-31T22:31:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:31:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:31:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-08-31T22:31:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:31:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 46 2024-08-31T22:31:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 46 2024-08-31T22:31:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:31:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=185, tag_tid=31, entry_tid=1], [object_number=184, tag_tid=31, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:31:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=185, tag_tid=31, entry_tid=1], [object_number=184, tag_tid=31, entry_tid=0], [object_number=187, tag_tid=29, entry_tid=7535], [object_number=186, tag_tid=29, entry_tid=7534]]], state=connected] 2024-08-31T22:31:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.status 2024-08-31T22:31:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=185, tag_tid=31, entry_tid=1' 2024-08-31T22:31:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=185, tag_tid=31, entry_tid=1' -a 'object_number=185, tag_tid=31, entry_tid=1' = 'object_number=185, tag_tid=31, entry_tid=1' 2024-08-31T22:31:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-08-31T22:31:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:31:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap10-clean 2024-08-31T22:31:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:31:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:31:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:31:08.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-08-31T22:31:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'test': 2024-08-31T22:31:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:31:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:31:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 20 2024-08-31T22:31:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a87b4c6a1b 2024-08-31T22:31:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a87b4c6a1b 2024-08-31T22:31:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:31:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:31:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:31:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:31:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:14:57 2024 2024-08-31T22:31:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:31:03 2024 2024-08-31T22:31:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:30:12 2024 2024-08-31T22:31:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:31:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a87b4c6a1b 2024-08-31T22:31:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:31:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:31:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: dd2c705e-25cf-4565-981f-d48de48eee84 2024-08-31T22:31:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:31:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 1 10 2024-08-31T22:31:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:31:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap1 2024-08-31T22:31:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap1 2024-08-31T22:31:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap1 2024-08-31T22:31:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:08.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap1-clean 2024-08-31T22:31:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap1-clean 2024-08-31T22:31:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap1-clean 2024-08-31T22:31:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:12.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:31:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap2 2024-08-31T22:31:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap2 2024-08-31T22:31:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap2 2024-08-31T22:31:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:14.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:14.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap2-clean 2024-08-31T22:31:14.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap2-clean 2024-08-31T22:31:14.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:14.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap2-clean 2024-08-31T22:31:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:14.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:31:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap3 2024-08-31T22:31:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap3 2024-08-31T22:31:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap3 2024-08-31T22:31:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:19.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap3-clean 2024-08-31T22:31:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap3-clean 2024-08-31T22:31:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap3-clean 2024-08-31T22:31:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:31:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap4 2024-08-31T22:31:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap4 2024-08-31T22:31:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap4 2024-08-31T22:31:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:22.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap4-clean 2024-08-31T22:31:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap4-clean 2024-08-31T22:31:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap4-clean 2024-08-31T22:31:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:31:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap5 2024-08-31T22:31:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap5 2024-08-31T22:31:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap5 2024-08-31T22:31:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:27.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:29.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap5-clean 2024-08-31T22:31:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap5-clean 2024-08-31T22:31:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap5-clean 2024-08-31T22:31:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:30.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:31.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:31:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap6 2024-08-31T22:31:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap6 2024-08-31T22:31:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap6 2024-08-31T22:31:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:32.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:32.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap6-clean 2024-08-31T22:31:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap6-clean 2024-08-31T22:31:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap6-clean 2024-08-31T22:31:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:31:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap7 2024-08-31T22:31:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap7 2024-08-31T22:31:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap7 2024-08-31T22:31:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap7-clean 2024-08-31T22:31:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap7-clean 2024-08-31T22:31:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap7-clean 2024-08-31T22:31:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:41.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:31:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap8 2024-08-31T22:31:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap8 2024-08-31T22:31:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap8 2024-08-31T22:31:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:41.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:41.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:41.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:41.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap8-clean 2024-08-31T22:31:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap8-clean 2024-08-31T22:31:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap8-clean 2024-08-31T22:31:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:45.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:31:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap9 2024-08-31T22:31:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap9 2024-08-31T22:31:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap9 2024-08-31T22:31:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap9-clean 2024-08-31T22:31:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap9-clean 2024-08-31T22:31:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap9-clean 2024-08-31T22:31:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:51.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:31:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap10 2024-08-31T22:31:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap10 2024-08-31T22:31:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap10 2024-08-31T22:31:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:52.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap10-clean 2024-08-31T22:31:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror test snap10-clean 2024-08-31T22:31:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap10-clean 2024-08-31T22:31:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:31:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export 2024-08-31T22:31:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-31T22:31:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-test.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-test.export 2024-08-31T22:31:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:31:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 1 10 2024-08-31T22:31:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:31:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap1 2024-08-31T22:31:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-08-31T22:31:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:31:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap1 2024-08-31T22:31:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-08-31T22:31:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:31:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap1-clean 2024-08-31T22:31:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-08-31T22:31:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:31:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap1-clean 2024-08-31T22:31:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-08-31T22:31:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:31:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:31:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap2 2024-08-31T22:31:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-08-31T22:31:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:31:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap2 2024-08-31T22:31:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-08-31T22:31:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:31:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap2-clean 2024-08-31T22:31:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-08-31T22:31:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:31:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:31:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:31:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap2-clean 2024-08-31T22:31:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-08-31T22:32:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:32:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:32:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap3 2024-08-31T22:32:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-08-31T22:32:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap3 2024-08-31T22:32:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-08-31T22:32:01.063 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:32:01.062+0000 7f338ab71640 -1 reset not still connected to 0x555ce28676c0 2024-08-31T22:32:01.063 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:32:01.062+0000 7f338ab71640 -1 reset not still connected to 0x555ce2867a00 2024-08-31T22:32:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:32:01.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap3-clean 2024-08-31T22:32:01.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-08-31T22:32:01.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:01.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:01.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:01.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap3-clean 2024-08-31T22:32:01.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-08-31T22:32:02.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:32:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:32:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap4 2024-08-31T22:32:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-08-31T22:32:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap4 2024-08-31T22:32:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-08-31T22:32:03.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:32:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap4-clean 2024-08-31T22:32:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-08-31T22:32:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap4-clean 2024-08-31T22:32:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-08-31T22:32:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:32:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:32:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap5 2024-08-31T22:32:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-08-31T22:32:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap5 2024-08-31T22:32:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-08-31T22:32:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:32:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap5-clean 2024-08-31T22:32:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-08-31T22:32:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap5-clean 2024-08-31T22:32:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-08-31T22:32:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:32:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:32:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap6 2024-08-31T22:32:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-08-31T22:32:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap6 2024-08-31T22:32:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-08-31T22:32:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:32:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap6-clean 2024-08-31T22:32:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-08-31T22:32:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap6-clean 2024-08-31T22:32:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-08-31T22:32:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:32:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:32:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap7 2024-08-31T22:32:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-08-31T22:32:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap7 2024-08-31T22:32:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-08-31T22:32:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:32:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap7-clean 2024-08-31T22:32:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-08-31T22:32:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap7-clean 2024-08-31T22:32:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-08-31T22:32:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:32:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:32:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap8 2024-08-31T22:32:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-08-31T22:32:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap8 2024-08-31T22:32:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-08-31T22:32:11.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:32:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap8-clean 2024-08-31T22:32:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-08-31T22:32:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap8-clean 2024-08-31T22:32:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-08-31T22:32:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:32:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:32:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap9 2024-08-31T22:32:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-08-31T22:32:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap9 2024-08-31T22:32:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-08-31T22:32:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:32:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap9-clean 2024-08-31T22:32:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-08-31T22:32:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap9-clean 2024-08-31T22:32:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-08-31T22:32:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:32:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:32:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap10 2024-08-31T22:32:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-08-31T22:32:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap10 2024-08-31T22:32:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-08-31T22:32:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:32:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap10-clean 2024-08-31T22:32:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-08-31T22:32:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap10-clean 2024-08-31T22:32:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-08-31T22:32:16.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:32:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror test 2024-08-31T22:32:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:32:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:32:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror test 2024-08-31T22:32:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-08-31T22:32:16.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-08-31T22:32:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-08-31T22:32:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:32:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-08-31T22:32:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:32:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:32:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:32:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:32:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:32:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror test 2024-08-31T22:32:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:32:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:32:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:32:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-08-31T22:32:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:32:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id=10a87b4c6a1b 2024-08-31T22:32:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:32:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:32:16.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror test deleted 10a87b4c6a1b 2024-08-31T22:32:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:32:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:32:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id=10a87b4c6a1b 2024-08-31T22:32:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:32:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:32:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror test 2024-08-31T22:32:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:32:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:32:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:32:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-08-31T22:32:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:32:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id=10a87b4c6a1b 2024-08-31T22:32:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 10a87b4c6a1b 2024-08-31T22:32:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -z 10a87b4c6a1b -o 10a87b4c6a1b = 10a87b4c6a1b 2024-08-31T22:32:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=present 2024-08-31T22:32:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = present 2024-08-31T22:32:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:32:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:32:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror test deleted 10a87b4c6a1b 2024-08-31T22:32:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:32:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=test 2024-08-31T22:32:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:32:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id=10a87b4c6a1b 2024-08-31T22:32:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:32:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:32:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror test 2024-08-31T22:32:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:32:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:32:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=test 2024-08-31T22:32:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-08-31T22:32:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:32:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image test: (2) No such file or directory 2024-08-31T22:32:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:32:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:32:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:32:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:32:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ testlog 'TEST: create many images' 2024-08-31T22:32:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ tee -a /tmp/tmp.iYfzsjLopX/rbd-mirror.test.log 2024-08-31T22:32:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ date '+%F %T' 2024-08-31T22:32:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo 2024-08-31 22:32:18 TEST: create many images 2024-08-31T22:32:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:2024-08-31 22:32:18 TEST: create many images 2024-08-31T22:32:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ snap_name=snap 2024-08-31T22:32:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 1 50 2024-08-31T22:32:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:32:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_1 2024-08-31T22:32:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 journal 128M 2024-08-31T22:32:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:32:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:32:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:32:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_1 128M 2024-08-31T22:32:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:32:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:32:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:32:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:32:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_1 2024-08-31T22:32:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:32:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:32:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:32:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:32:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:32:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_1 2024-08-31T22:32:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:32:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:32:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=14 2024-08-31T22:32:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:32:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:32:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 31458304 bytes 2024-08-31T22:32:22.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 73521152 bytes 2024-08-31T22:32:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 165424128 bytes 2024-08-31T22:32:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 184110080 bytes 2024-08-31T22:32:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:500 IOs, 202846208 bytes 2024-08-31T22:32:32.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:32:32.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:32:32.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:32:32.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:32:32.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:32:32.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_2 2024-08-31T22:32:32.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 journal 128M 2024-08-31T22:32:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:32:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:32:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:32:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_2 128M 2024-08-31T22:32:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:32:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:32:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:32:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:32:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_2 2024-08-31T22:32:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:32:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:32:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:32:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:32:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:32:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_2 2024-08-31T22:32:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:32:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:32:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=8 2024-08-31T22:32:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:32:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:32:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 5136384 bytes 2024-08-31T22:32:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 21695488 bytes 2024-08-31T22:32:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 89107456 bytes 2024-08-31T22:32:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:32:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:32:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:32:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:32:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:32:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_3 2024-08-31T22:32:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 journal 128M 2024-08-31T22:32:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:32:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:32:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:32:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_3 128M 2024-08-31T22:32:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:32:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:32:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:32:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:32:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_3 2024-08-31T22:32:40.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:32:40.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:40.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:32:40.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:32:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:32:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:32:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_3 2024-08-31T22:32:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:32:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:32:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=13 2024-08-31T22:32:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:32:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:32:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 107402240 bytes 2024-08-31T22:32:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 124414976 bytes 2024-08-31T22:32:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 178209792 bytes 2024-08-31T22:32:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:32:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:32:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:32:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:32:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:32:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_4 2024-08-31T22:32:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 journal 128M 2024-08-31T22:32:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:32:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:32:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:32:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_4 128M 2024-08-31T22:32:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:32:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:32:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:32:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:32:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:32:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_4 2024-08-31T22:32:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:32:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:32:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:32:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:32:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:32:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_4 2024-08-31T22:32:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:32:53.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:32:53.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:32:53.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:32:53.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=10 2024-08-31T22:32:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:32:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:32:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 10691584 bytes 2024-08-31T22:32:55.060 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:32:55.061+0000 7f338ab71640 -1 reset not still connected to 0x555ce162d110 2024-08-31T22:32:55.060 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:32:55.061+0000 7f338ab71640 -1 reset not still connected to 0x555ce2638a90 2024-08-31T22:32:55.062 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:32:55.065+0000 7f338ab71640 -1 reset not still connected to 0x555ce26392b0 2024-08-31T22:32:55.062 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:32:55.065+0000 7f338ab71640 -1 reset not still connected to 0x555ce2866000 2024-08-31T22:32:55.062 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:32:55.065+0000 7f338ab71640 -1 reset not still connected to 0x555ce293a410 2024-08-31T22:32:55.063 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:32:55.065+0000 7f338ab71640 -1 reset not still connected to 0x555ce66a0dd0 2024-08-31T22:32:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 109356032 bytes 2024-08-31T22:33:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 154259456 bytes 2024-08-31T22:33:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:33:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:33:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:33:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:33:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:33:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_5 2024-08-31T22:33:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 journal 128M 2024-08-31T22:33:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:33:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:33:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:33:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_5 128M 2024-08-31T22:33:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:33:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:33:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:33:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:33:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_5 2024-08-31T22:33:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:33:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:33:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:33:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:33:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:33:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_5 2024-08-31T22:33:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:33:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:33:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=5 2024-08-31T22:33:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:33:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:33:04.286 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:04.285+0000 7f338ab71640 -1 reset not still connected to 0x555ce26391e0 2024-08-31T22:33:04.286 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:04.285+0000 7f338ab71640 -1 reset not still connected to 0x555ce28676c0 2024-08-31T22:33:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 47862784 bytes 2024-08-31T22:33:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:33:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:33:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:33:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:33:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:33:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_6 2024-08-31T22:33:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 journal 128M 2024-08-31T22:33:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:33:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:33:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:33:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_6 128M 2024-08-31T22:33:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:33:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:33:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:33:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:33:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_6 2024-08-31T22:33:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:33:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:33:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:33:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:33:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:33:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_6 2024-08-31T22:33:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:33:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:33:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=5 2024-08-31T22:33:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:33:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:33:10.661 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:33:10.661+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff161a0 2024-08-31T22:33:10.661 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:33:10.661+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff169c0 2024-08-31T22:33:10.661 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:33:10.661+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff16d00 2024-08-31T22:33:10.661 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:33:10.665+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff17c70 2024-08-31T22:33:10.709 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:33:10.709+0000 7f13efcb9640 -1 reset not still connected to 0x559f1de35380 2024-08-31T22:33:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:33:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:33:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:33:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:33:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:33:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_7 2024-08-31T22:33:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 journal 128M 2024-08-31T22:33:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:33:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:33:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:33:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_7 128M 2024-08-31T22:33:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:33:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:33:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:33:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:33:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_7 2024-08-31T22:33:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:33:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:33:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:33:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:33:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:33:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_7 2024-08-31T22:33:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:33:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:33:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=7 2024-08-31T22:33:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:33:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:33:13.737 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:13.737+0000 7f338ab71640 -1 reset not still connected to 0x555ce26392b0 2024-08-31T22:33:13.737 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:13.737+0000 7f338ab71640 -1 reset not still connected to 0x555ce2866000 2024-08-31T22:33:13.737 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:13.737+0000 7f338ab71640 -1 reset not still connected to 0x555ce293a410 2024-08-31T22:33:13.737 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:13.737+0000 7f338ab71640 -1 reset not still connected to 0x555ce66a0dd0 2024-08-31T22:33:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 16756736 bytes 2024-08-31T22:33:14.905 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:14.905+0000 7f338ab71640 -1 reset not still connected to 0x555ce133fa00 2024-08-31T22:33:14.905 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:14.905+0000 7f338ab71640 -1 reset not still connected to 0x555ce162d110 2024-08-31T22:33:14.905 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:14.905+0000 7f338ab71640 -1 reset not still connected to 0x555ce2638a90 2024-08-31T22:33:14.906 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:14.905+0000 7f338ab71640 -1 reset not still connected to 0x555ce26392b0 2024-08-31T22:33:14.906 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:14.905+0000 7f338ab71640 -1 reset not still connected to 0x555ce2866000 2024-08-31T22:33:14.906 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:14.905+0000 7f338ab71640 -1 reset not still connected to 0x555ce293a410 2024-08-31T22:33:14.906 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:14.905+0000 7f338ab71640 -1 reset not still connected to 0x555ce66a0dd0 2024-08-31T22:33:17.191 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:33:17.189+0000 7ff273ffe640 -1 reset not still connected to 0x561947f8c0d0 2024-08-31T22:33:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 67502080 bytes 2024-08-31T22:33:18.310 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:18.313+0000 7f6588f42640 -1 reset not still connected to 0x56202499a0d0 2024-08-31T22:33:18.310 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:18.313+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:33:18.310 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:18.313+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:33:18.310 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:18.313+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:33:18.311 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:18.313+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:33:18.397 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:18.397+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:33:18.397 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:18.397+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:33:18.459 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:18.461+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:33:18.459 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:18.461+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:33:18.459 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:18.461+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:33:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:33:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:33:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:33:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:33:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:33:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_8 2024-08-31T22:33:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 journal 128M 2024-08-31T22:33:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:33:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:33:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:33:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_8 128M 2024-08-31T22:33:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:33:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:33:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:33:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:33:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_8 2024-08-31T22:33:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:33:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:33:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:33:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:33:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:33:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_8 2024-08-31T22:33:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:33:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:33:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=7 2024-08-31T22:33:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:33:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:33:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 43803648 bytes 2024-08-31T22:33:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 111318016 bytes 2024-08-31T22:33:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:33:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:33:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:33:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:33:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:33:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_9 2024-08-31T22:33:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 journal 128M 2024-08-31T22:33:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:33:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:33:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:33:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_9 128M 2024-08-31T22:33:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:33:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:33:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:33:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:33:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_9 2024-08-31T22:33:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:33:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:33:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:33:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:33:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:33:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_9 2024-08-31T22:33:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:33:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:33:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=13 2024-08-31T22:33:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:33:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:33:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 32305152 bytes 2024-08-31T22:33:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 32925696 bytes 2024-08-31T22:33:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 82788352 bytes 2024-08-31T22:33:37.262 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:37.260+0000 7f6588f42640 -1 reset not still connected to 0x56202461dd40 2024-08-31T22:33:37.263 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:37.264+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:33:37.263 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:37.264+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:33:37.264 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:37.264+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:33:37.264 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:37.264+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:33:37.264 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:37.264+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:33:37.264 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:37.264+0000 7f6588f42640 -1 reset not still connected to 0x56202461dd40 2024-08-31T22:33:37.264 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:37.264+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:33:37.264 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:37.264+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:33:37.264 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:37.264+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:33:37.264 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:33:37.264+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:33:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 129409024 bytes 2024-08-31T22:33:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:500 IOs, 157524992 bytes 2024-08-31T22:33:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:33:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:33:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:33:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:33:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:33:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_10 2024-08-31T22:33:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 journal 128M 2024-08-31T22:33:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:33:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:33:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:33:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_10 128M 2024-08-31T22:33:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:33:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:33:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:33:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:33:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_10 2024-08-31T22:33:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:33:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:33:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:33:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:33:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:33:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_10 2024-08-31T22:33:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:33:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:33:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=11 2024-08-31T22:33:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:33:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:33:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 618496 bytes 2024-08-31T22:33:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 19410944 bytes 2024-08-31T22:33:44.386 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:44.384+0000 7f338ab71640 -1 reset not still connected to 0x555ce133fa00 2024-08-31T22:33:44.386 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:44.384+0000 7f338ab71640 -1 reset not still connected to 0x555ce162d110 2024-08-31T22:33:44.387 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:44.384+0000 7f338ab71640 -1 reset not still connected to 0x555ce2638a90 2024-08-31T22:33:44.387 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:44.384+0000 7f338ab71640 -1 reset not still connected to 0x555ce26392b0 2024-08-31T22:33:44.387 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:44.384+0000 7f338ab71640 -1 reset not still connected to 0x555ce2866000 2024-08-31T22:33:44.387 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:44.384+0000 7f338ab71640 -1 reset not still connected to 0x555ce293a410 2024-08-31T22:33:44.387 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:44.384+0000 7f338ab71640 -1 reset not still connected to 0x555ce66a0dd0 2024-08-31T22:33:49.054 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:49.052+0000 7f338ab71640 -1 reset not still connected to 0x555ce2638a90 2024-08-31T22:33:49.055 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:49.052+0000 7f338ab71640 -1 reset not still connected to 0x555ce26392b0 2024-08-31T22:33:49.055 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:49.052+0000 7f338ab71640 -1 reset not still connected to 0x555ce2866000 2024-08-31T22:33:49.055 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:49.052+0000 7f338ab71640 -1 reset not still connected to 0x555ce293a410 2024-08-31T22:33:49.055 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:49.052+0000 7f338ab71640 -1 reset not still connected to 0x555ce66a0dd0 2024-08-31T22:33:49.055 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:49.052+0000 7f338ab71640 -1 reset not still connected to 0x555ce133e820 2024-08-31T22:33:49.055 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:49.056+0000 7f338ab71640 -1 reset not still connected to 0x555ce133fa00 2024-08-31T22:33:49.055 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:33:49.056+0000 7f338ab71640 -1 reset not still connected to 0x555ce66a0dd0 2024-08-31T22:33:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 129094656 bytes 2024-08-31T22:33:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:33:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:33:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:33:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:33:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:33:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_11 2024-08-31T22:33:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 journal 128M 2024-08-31T22:33:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_11 2024-08-31T22:33:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:33:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:33:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_11 128M 2024-08-31T22:33:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_11 2024-08-31T22:33:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:33:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:33:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:33:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:33:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_11 2024-08-31T22:33:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:33:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:33:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:33:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:33:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:33:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_11 2024-08-31T22:33:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:33:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:33:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_11 2024-08-31T22:33:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:33:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=9 2024-08-31T22:33:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:33:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:33:57.884 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:33:57.884+0000 7ff273ffe640 -1 reset not still connected to 0x56193c12d520 2024-08-31T22:33:57.884 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:33:57.884+0000 7ff273ffe640 -1 reset not still connected to 0x561947f30b60 2024-08-31T22:33:57.884 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:33:57.884+0000 7ff273ffe640 -1 reset not still connected to 0x56194f1f8b60 2024-08-31T22:33:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 89825280 bytes 2024-08-31T22:34:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:34:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:34:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:34:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:34:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:34:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_12 2024-08-31T22:34:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 journal 128M 2024-08-31T22:34:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_12 2024-08-31T22:34:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:34:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:34:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_12 128M 2024-08-31T22:34:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_12 2024-08-31T22:34:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:34:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:34:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:34:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_12 2024-08-31T22:34:01.009 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:34:01.012+0000 7ff273ffe640 -1 reset not still connected to 0x56193301c8f0 2024-08-31T22:34:01.009 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:34:01.012+0000 7ff273ffe640 -1 reset not still connected to 0x56194cb5b520 2024-08-31T22:34:01.009 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:34:01.012+0000 7ff273ffe640 -1 reset not still connected to 0x56194e533ad0 2024-08-31T22:34:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:34:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:34:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:34:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:34:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:34:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_12 2024-08-31T22:34:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_12 2024-08-31T22:34:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:34:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=5 2024-08-31T22:34:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:34:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:34:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:34:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:34:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:34:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:34:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:34:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_13 2024-08-31T22:34:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 journal 128M 2024-08-31T22:34:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_13 2024-08-31T22:34:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:34:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:34:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_13 128M 2024-08-31T22:34:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_13 2024-08-31T22:34:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:34:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:34:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:34:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:06.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_13 2024-08-31T22:34:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:34:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:34:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:34:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:34:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:34:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_13 2024-08-31T22:34:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_13 2024-08-31T22:34:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:34:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=13 2024-08-31T22:34:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:34:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:34:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 10692608 bytes 2024-08-31T22:34:08.240 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:34:08.240+0000 7f1a77b11640 -1 reset not still connected to 0x5649c0c96b60 2024-08-31T22:34:08.241 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:34:08.240+0000 7f1a77b11640 -1 reset not still connected to 0x5649d7389c70 2024-08-31T22:34:08.241 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:34:08.240+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:34:08.241 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:34:08.240+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:34:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 52676608 bytes 2024-08-31T22:34:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 167548928 bytes 2024-08-31T22:34:17.935 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:17.936+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:34:17.937 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:17.936+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:34:17.937 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:17.936+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:34:17.938 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:17.936+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:34:17.938 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:17.936+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:34:17.938 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:17.936+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:34:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:34:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:34:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:34:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:34:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:34:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_14 2024-08-31T22:34:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 journal 128M 2024-08-31T22:34:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_14 2024-08-31T22:34:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:34:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:34:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_14 128M 2024-08-31T22:34:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_14 2024-08-31T22:34:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:34:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:34:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:34:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_14 2024-08-31T22:34:19.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:34:19.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:19.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:34:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:34:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:34:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:34:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_14 2024-08-31T22:34:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_14 2024-08-31T22:34:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:34:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=7 2024-08-31T22:34:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:34:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:34:20.385 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:34:20.380+0000 7f338ab71640 -1 reset not still connected to 0x555ce133fe10 2024-08-31T22:34:20.385 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:34:20.380+0000 7f338ab71640 -1 reset not still connected to 0x555ce26391e0 2024-08-31T22:34:20.388 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:34:20.388+0000 7f338ab71640 -1 reset not still connected to 0x555ce28676c0 2024-08-31T22:34:20.390 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:34:20.392+0000 7f338ab71640 -1 reset not still connected to 0x555ce2867a00 2024-08-31T22:34:20.390 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:34:20.392+0000 7f338ab71640 -1 reset not still connected to 0x555ce58b2b60 2024-08-31T22:34:20.390 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:34:20.392+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:34:20.390 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:34:20.392+0000 7f338ab71640 -1 reset not still connected to 0x555cf25991e0 2024-08-31T22:34:20.390 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:34:20.392+0000 7f338ab71640 -1 reset not still connected to 0x555d134d2000 2024-08-31T22:34:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 23567360 bytes 2024-08-31T22:34:22.320 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:22.320+0000 7f6588f42640 -1 reset not still connected to 0x56202499a0d0 2024-08-31T22:34:22.320 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:22.320+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:34:22.320 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:22.320+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:34:22.320 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:22.320+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:34:22.320 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:22.320+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:34:22.320 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:22.320+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:34:22.320 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:22.320+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:34:23.046 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:23.044+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:34:23.046 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:23.044+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:34:23.046 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:23.044+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:34:23.046 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:23.044+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:34:23.046 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:23.044+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:34:23.046 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:23.044+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:34:23.047 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:23.048+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:34:23.047 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:23.048+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:34:23.047 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:23.048+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:34:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 44876800 bytes 2024-08-31T22:34:25.195 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:25.191+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:34:25.195 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:25.191+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:34:25.195 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:25.191+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:34:25.323 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:25.323+0000 7f6588f42640 -1 reset not still connected to 0x56202997c9c0 2024-08-31T22:34:25.323 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:25.323+0000 7f6588f42640 -1 reset not still connected to 0x5620326af930 2024-08-31T22:34:25.325 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:25.323+0000 7f6588f42640 -1 reset not still connected to 0x562032a40680 2024-08-31T22:34:25.326 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:25.327+0000 7f6588f42640 -1 reset not still connected to 0x562032a40d00 2024-08-31T22:34:25.326 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:25.327+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:34:25.326 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:25.327+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:34:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:34:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:34:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:34:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:34:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:34:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_15 2024-08-31T22:34:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 journal 128M 2024-08-31T22:34:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_15 2024-08-31T22:34:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:34:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:34:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_15 128M 2024-08-31T22:34:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_15 2024-08-31T22:34:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:34:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:34:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:34:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_15 2024-08-31T22:34:26.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:34:26.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:26.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:34:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:34:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:34:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:34:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_15 2024-08-31T22:34:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_15 2024-08-31T22:34:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:34:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=11 2024-08-31T22:34:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:34:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:34:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 44581888 bytes 2024-08-31T22:34:28.506 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:28.503+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:34:28.506 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:28.503+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:34:28.506 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:28.503+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:34:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 52961280 bytes 2024-08-31T22:34:31.363 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:31.363+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:34:31.363 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:31.363+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:34:31.363 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:31.363+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:34:31.363 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:31.363+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:34:31.805 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:31.803+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:34:31.805 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:31.803+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:34:31.806 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:31.803+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:34:31.812 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:31.811+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:34:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 105286656 bytes 2024-08-31T22:34:34.932 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:34:34.932+0000 7f13efcb9640 -1 reset not still connected to 0x559f2b7404e0 2024-08-31T22:34:35.958 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:34:35.960+0000 7f13efcb9640 -1 reset not still connected to 0x559f14700b60 2024-08-31T22:34:35.959 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:34:35.960+0000 7f13efcb9640 -1 reset not still connected to 0x559f184a0270 2024-08-31T22:34:35.960 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:34:35.960+0000 7f13efcb9640 -1 reset not still connected to 0x559f19a8ed00 2024-08-31T22:34:35.961 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:34:35.960+0000 7f13efcb9640 -1 reset not still connected to 0x559f2b7404e0 2024-08-31T22:34:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:34:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:34:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:34:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:34:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:34:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_16 2024-08-31T22:34:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 journal 128M 2024-08-31T22:34:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_16 2024-08-31T22:34:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:34:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:34:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_16 128M 2024-08-31T22:34:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_16 2024-08-31T22:34:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:34:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:34:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:34:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_16 2024-08-31T22:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_16 2024-08-31T22:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_16 2024-08-31T22:34:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:34:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=6 2024-08-31T22:34:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:34:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:34:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 23717888 bytes 2024-08-31T22:34:39.053 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:34:39.051+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:34:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 64153600 bytes 2024-08-31T22:34:42.382 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:34:42.379+0000 7f338ab71640 -1 reset not still connected to 0x555ce2638a90 2024-08-31T22:34:42.382 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:34:42.379+0000 7f338ab71640 -1 reset not still connected to 0x555ce26392b0 2024-08-31T22:34:42.382 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:34:42.379+0000 7f338ab71640 -1 reset not still connected to 0x555ce2866000 2024-08-31T22:34:42.382 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:34:42.379+0000 7f338ab71640 -1 reset not still connected to 0x555ce293a410 2024-08-31T22:34:42.382 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:34:42.379+0000 7f338ab71640 -1 reset not still connected to 0x555ce66a0dd0 2024-08-31T22:34:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:34:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:34:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:34:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:34:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:34:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_17 2024-08-31T22:34:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 journal 128M 2024-08-31T22:34:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_17 2024-08-31T22:34:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:34:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:34:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_17 128M 2024-08-31T22:34:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_17 2024-08-31T22:34:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:34:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:34:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:34:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_17 2024-08-31T22:34:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:34:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:34:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:34:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:34:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:34:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_17 2024-08-31T22:34:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_17 2024-08-31T22:34:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:34:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=10 2024-08-31T22:34:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:34:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:34:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 50300928 bytes 2024-08-31T22:34:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 76457984 bytes 2024-08-31T22:34:51.409 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:34:51.411+0000 7ff273ffe640 -1 reset not still connected to 0x561947f30b60 2024-08-31T22:34:51.409 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:34:51.411+0000 7ff273ffe640 -1 reset not still connected to 0x56194f1f8b60 2024-08-31T22:34:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 92800000 bytes 2024-08-31T22:34:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:34:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:34:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:34:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:34:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:34:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_18 2024-08-31T22:34:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 journal 128M 2024-08-31T22:34:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_18 2024-08-31T22:34:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:34:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:34:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_18 128M 2024-08-31T22:34:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_18 2024-08-31T22:34:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:34:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:34:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:34:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:34:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_18 2024-08-31T22:34:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:34:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:34:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:34:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:34:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:34:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_18 2024-08-31T22:34:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:34:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:34:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_18 2024-08-31T22:34:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:34:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=7 2024-08-31T22:34:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:34:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:34:55.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 19184640 bytes 2024-08-31T22:35:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:35:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:35:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:35:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:35:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:35:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_19 2024-08-31T22:35:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 journal 128M 2024-08-31T22:35:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:35:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_19 2024-08-31T22:35:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:35:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:35:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_19 128M 2024-08-31T22:35:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:35:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_19 2024-08-31T22:35:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:35:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:35:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:35:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_19 2024-08-31T22:35:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:35:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:35:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:35:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:35:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:35:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_19 2024-08-31T22:35:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:35:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_19 2024-08-31T22:35:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:35:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=13 2024-08-31T22:35:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:35:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:35:01.140 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:01.135+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:35:01.220 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:01.135+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:35:01.221 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:01.135+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:35:01.221 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:01.135+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:35:01.221 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:01.135+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:35:01.221 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:01.135+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:35:01.221 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:01.135+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:35:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 59324416 bytes 2024-08-31T22:35:03.803 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:03.803+0000 7f6588f42640 -1 reset not still connected to 0x56202499a0d0 2024-08-31T22:35:03.803 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:03.803+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:35:03.803 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:03.803+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:35:03.803 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:03.803+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:35:03.803 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:03.803+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:35:03.803 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:03.803+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:35:03.804 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:03.803+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:35:03.804 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:03.803+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:35:03.804 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:03.803+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:35:03.804 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:03.803+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:35:03.804 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:03.803+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:35:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 79655936 bytes 2024-08-31T22:35:07.498 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:07.499+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:35:07.500 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:07.499+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:35:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 100807680 bytes 2024-08-31T22:35:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 118142976 bytes 2024-08-31T22:35:09.674 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:35:09.675+0000 7f1a77b11640 -1 reset not still connected to 0x5649c0c96b60 2024-08-31T22:35:09.675 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:35:09.675+0000 7f1a77b11640 -1 reset not still connected to 0x5649d7389c70 2024-08-31T22:35:09.675 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:35:09.675+0000 7f1a77b11640 -1 reset not still connected to 0x5649dca20680 2024-08-31T22:35:09.675 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:35:09.675+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:35:09.675 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:35:09.675+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:35:09.675 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:35:09.675+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:35:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:500 IOs, 191742976 bytes 2024-08-31T22:35:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:600 IOs, 196653056 bytes 2024-08-31T22:35:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:35:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:35:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:35:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:35:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:35:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_20 2024-08-31T22:35:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 journal 128M 2024-08-31T22:35:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:35:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_20 2024-08-31T22:35:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:35:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:35:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_20 128M 2024-08-31T22:35:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:35:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_20 2024-08-31T22:35:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:35:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:35:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:35:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_20 2024-08-31T22:35:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:35:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:35:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:35:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:35:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:35:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_20 2024-08-31T22:35:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:35:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_20 2024-08-31T22:35:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:35:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=11 2024-08-31T22:35:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:35:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:35:16.020 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:16.019+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:35:16.021 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:16.019+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:35:16.021 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:16.019+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:35:16.024 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:16.023+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:35:16.024 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:16.023+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:35:16.024 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:16.023+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:35:16.024 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:16.023+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:35:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 68595712 bytes 2024-08-31T22:35:20.500 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:20.499+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:35:20.500 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:20.499+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:35:20.500 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:20.499+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:35:20.500 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:20.499+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:35:20.500 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:20.499+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:35:20.500 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:20.499+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:35:20.500 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:20.499+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:35:20.500 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:20.499+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:35:20.501 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:20.499+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:35:20.501 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:20.499+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:35:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 83149824 bytes 2024-08-31T22:35:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 128884736 bytes 2024-08-31T22:35:23.949 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:23.943+0000 7f6588f42640 -1 reset not still connected to 0x56202461dd40 2024-08-31T22:35:23.949 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:23.947+0000 7f6588f42640 -1 reset not still connected to 0x56202499a0d0 2024-08-31T22:35:23.949 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:23.947+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:35:23.949 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:23.947+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:35:23.949 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:23.947+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:35:23.949 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:23.947+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:35:23.949 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:23.947+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:35:23.949 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:23.947+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:35:23.949 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:23.947+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:35:23.949 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:23.947+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:35:23.949 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:23.947+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:35:23.949 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:23.947+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:35:24.367 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:35:24.367+0000 7f1a77b11640 -1 reset not still connected to 0x5649dca20680 2024-08-31T22:35:24.367 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:35:24.367+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:35:24.367 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:35:24.367+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:35:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:35:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:35:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:35:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:35:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:35:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_21 2024-08-31T22:35:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 journal 128M 2024-08-31T22:35:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:35:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_21 2024-08-31T22:35:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:35:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:35:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_21 128M 2024-08-31T22:35:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:35:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_21 2024-08-31T22:35:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:35:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:35:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:35:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_21 2024-08-31T22:35:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:35:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:35:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:35:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:35:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:35:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_21 2024-08-31T22:35:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:35:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_21 2024-08-31T22:35:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:35:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=5 2024-08-31T22:35:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:35:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:35:27.296 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:35:27.294+0000 7f338ab71640 -1 reset not still connected to 0x555ce26392b0 2024-08-31T22:35:27.297 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:35:27.294+0000 7f338ab71640 -1 reset not still connected to 0x555ce293a410 2024-08-31T22:35:27.297 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:35:27.294+0000 7f338ab71640 -1 reset not still connected to 0x555ce66a0dd0 2024-08-31T22:35:27.304 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:35:27.302+0000 7f338ab71640 -1 reset not still connected to 0x555d0a5768f0 2024-08-31T22:35:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 40397824 bytes 2024-08-31T22:35:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:35:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:35:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:35:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:35:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:35:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_22 2024-08-31T22:35:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 journal 128M 2024-08-31T22:35:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:35:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_22 2024-08-31T22:35:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:35:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:35:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_22 128M 2024-08-31T22:35:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:35:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_22 2024-08-31T22:35:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:35:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:35:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:35:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_22 2024-08-31T22:35:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:35:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:35:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:35:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:35:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:35:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_22 2024-08-31T22:35:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:35:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_22 2024-08-31T22:35:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:35:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=13 2024-08-31T22:35:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:35:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:35:31.760 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:35:31.759+0000 7f13efcb9640 -1 reset not still connected to 0x559f2aed8dd0 2024-08-31T22:35:31.848 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:35:31.842+0000 7f338ab71640 -1 reset not still connected to 0x555ce133fe10 2024-08-31T22:35:31.854 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:35:31.850+0000 7f338ab71640 -1 reset not still connected to 0x555ce26391e0 2024-08-31T22:35:31.854 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:35:31.850+0000 7f338ab71640 -1 reset not still connected to 0x555ce2673450 2024-08-31T22:35:31.854 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:35:31.850+0000 7f338ab71640 -1 reset not still connected to 0x555ce28676c0 2024-08-31T22:35:31.854 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:35:31.850+0000 7f338ab71640 -1 reset not still connected to 0x555ce2867a00 2024-08-31T22:35:31.854 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:35:31.850+0000 7f338ab71640 -1 reset not still connected to 0x555ce58b2b60 2024-08-31T22:35:31.854 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:35:31.850+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:35:31.854 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:35:31.850+0000 7f338ab71640 -1 reset not still connected to 0x555cf25991e0 2024-08-31T22:35:31.854 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:35:31.850+0000 7f338ab71640 -1 reset not still connected to 0x555cf7f8f450 2024-08-31T22:35:31.855 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:35:31.850+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:35:31.855 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:35:31.850+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:35:31.855 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:35:31.850+0000 7f338ab71640 -1 reset not still connected to 0x555d134d2000 2024-08-31T22:35:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 22716416 bytes 2024-08-31T22:35:34.935 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:34.934+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:35:34.935 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:34.934+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:35:34.935 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:34.934+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:35:34.936 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:34.934+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:35:34.936 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:34.934+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:35:34.936 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:34.934+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:35:34.936 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:34.934+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:35:34.939 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:34.938+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:35:34.939 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:34.938+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:35:34.939 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:34.938+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:35:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 107830272 bytes 2024-08-31T22:35:39.746 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:39.746+0000 7f6588f42640 -1 reset not still connected to 0x562032a40d00 2024-08-31T22:35:39.747 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:39.746+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:35:39.747 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:39.746+0000 7f6588f42640 -1 reset not still connected to 0x562045dee270 2024-08-31T22:35:39.747 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:39.746+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:35:39.747 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:39.746+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:35:39.803 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:39.802+0000 7f6588f42640 -1 reset not still connected to 0x562032a40d00 2024-08-31T22:35:39.803 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:39.802+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:35:39.803 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:39.802+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:35:39.803 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:39.802+0000 7f6588f42640 -1 reset not still connected to 0x562045dee270 2024-08-31T22:35:39.803 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:39.802+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:35:39.803 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:39.802+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:35:39.803 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:39.802+0000 7f6588f42640 -1 reset not still connected to 0x562028e1b520 2024-08-31T22:35:39.804 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:39.802+0000 7f6588f42640 -1 reset not still connected to 0x56202997c9c0 2024-08-31T22:35:39.804 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:39.802+0000 7f6588f42640 -1 reset not still connected to 0x5620326af930 2024-08-31T22:35:39.808 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:39.806+0000 7f6588f42640 -1 reset not still connected to 0x562032a40680 2024-08-31T22:35:39.809 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:39.806+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:35:39.809 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:39.806+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:35:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 149075968 bytes 2024-08-31T22:35:41.418 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:35:41.418+0000 7ff273ffe640 -1 reset not still connected to 0x56194e533ad0 2024-08-31T22:35:41.418 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:35:41.418+0000 7ff273ffe640 -1 reset not still connected to 0x5619515f5110 2024-08-31T22:35:41.583 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:35:41.582+0000 7ff273ffe640 -1 reset not still connected to 0x56194e533ad0 2024-08-31T22:35:41.584 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:35:41.582+0000 7ff273ffe640 -1 reset not still connected to 0x5619515f5110 2024-08-31T22:35:43.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 168947712 bytes 2024-08-31T22:35:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:35:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:35:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:35:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:35:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:35:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_23 2024-08-31T22:35:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 journal 128M 2024-08-31T22:35:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:35:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_23 2024-08-31T22:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_23 128M 2024-08-31T22:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_23 2024-08-31T22:35:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:35:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:35:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:35:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_23 2024-08-31T22:35:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:35:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:35:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:35:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:35:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:35:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_23 2024-08-31T22:35:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:35:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_23 2024-08-31T22:35:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:35:43.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=11 2024-08-31T22:35:43.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:35:43.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:35:44.933 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:44.930+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:35:44.933 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:44.930+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:35:44.942 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:44.942+0000 7f6588f42640 -1 reset not still connected to 0x5620326af930 2024-08-31T22:35:44.942 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:44.942+0000 7f6588f42640 -1 reset not still connected to 0x562032a40680 2024-08-31T22:35:44.943 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:44.942+0000 7f6588f42640 -1 reset not still connected to 0x562032a40d00 2024-08-31T22:35:44.943 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:44.942+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:35:44.950 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:44.946+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:35:44.950 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:44.946+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:35:44.950 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:44.946+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:35:45.205 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:35:45.206+0000 7ff273ffe640 -1 reset not still connected to 0x56194e533ad0 2024-08-31T22:35:45.205 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:35:45.206+0000 7ff273ffe640 -1 reset not still connected to 0x5619515f5110 2024-08-31T22:35:45.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 43087872 bytes 2024-08-31T22:35:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 75939840 bytes 2024-08-31T22:35:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 140595200 bytes 2024-08-31T22:35:52.980 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:52.974+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:35:52.980 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:52.974+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:35:52.980 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:52.974+0000 7f6588f42640 -1 reset not still connected to 0x562045dee270 2024-08-31T22:35:52.980 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:52.978+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:35:52.980 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:35:52.978+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:35:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:35:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:35:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:35:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:35:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:35:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_24 2024-08-31T22:35:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 journal 128M 2024-08-31T22:35:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:35:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_24 2024-08-31T22:35:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:35:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:35:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_24 128M 2024-08-31T22:35:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:35:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_24 2024-08-31T22:35:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:35:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:35:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:35:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:35:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_24 2024-08-31T22:35:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:35:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:35:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:35:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:35:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:35:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_24 2024-08-31T22:35:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:35:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:35:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_24 2024-08-31T22:35:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:35:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=13 2024-08-31T22:35:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:35:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:35:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 15560704 bytes 2024-08-31T22:35:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 65495040 bytes 2024-08-31T22:36:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 108381184 bytes 2024-08-31T22:36:05.390 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:05.390+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:36:05.394 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:05.398+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:36:05.394 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:05.398+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:36:05.395 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:05.398+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:36:05.395 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:05.398+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:36:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 171123712 bytes 2024-08-31T22:36:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:36:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:36:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:36:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:36:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:36:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_25 2024-08-31T22:36:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 journal 128M 2024-08-31T22:36:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:36:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_25 2024-08-31T22:36:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:36:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:36:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_25 128M 2024-08-31T22:36:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:36:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_25 2024-08-31T22:36:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:36:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:36:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:36:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_25 2024-08-31T22:36:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:36:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:36:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:36:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:36:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:36:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_25 2024-08-31T22:36:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:36:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_25 2024-08-31T22:36:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:36:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=5 2024-08-31T22:36:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:36:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:36:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 51380224 bytes 2024-08-31T22:36:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:36:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:36:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:36:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:36:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:36:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_26 2024-08-31T22:36:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 journal 128M 2024-08-31T22:36:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:36:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_26 2024-08-31T22:36:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:36:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:36:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_26 128M 2024-08-31T22:36:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:36:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_26 2024-08-31T22:36:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:36:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:36:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:36:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_26 2024-08-31T22:36:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:36:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:36:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:36:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:36:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:36:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_26 2024-08-31T22:36:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:36:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_26 2024-08-31T22:36:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:36:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=8 2024-08-31T22:36:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:36:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:36:14.824 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:14.825+0000 7f338ab71640 -1 reset not still connected to 0x555cf25991e0 2024-08-31T22:36:14.826 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:14.829+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:36:14.826 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:14.829+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:36:14.832 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:14.833+0000 7f338ab71640 -1 reset not still connected to 0x555d0eeeaa90 2024-08-31T22:36:14.832 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:14.833+0000 7f338ab71640 -1 reset not still connected to 0x555d134d2000 2024-08-31T22:36:15.804 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:15.805+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:36:15.804 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:15.805+0000 7f6588f42640 -1 reset not still connected to 0x5620404e49c0 2024-08-31T22:36:15.804 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:15.805+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:36:15.804 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:15.805+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:36:15.804 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:15.805+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:36:15.804 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:15.805+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:36:15.804 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:15.805+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:36:15.804 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:15.805+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:36:15.804 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:15.805+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:36:15.804 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:15.805+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:36:15.805 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:15.805+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:36:16.237 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:16.237+0000 7f338ab71640 -1 reset not still connected to 0x555ce133fe10 2024-08-31T22:36:16.238 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:16.237+0000 7f338ab71640 -1 reset not still connected to 0x555ce26391e0 2024-08-31T22:36:16.246 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:16.245+0000 7f338ab71640 -1 reset not still connected to 0x555ce2673450 2024-08-31T22:36:16.246 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:16.245+0000 7f338ab71640 -1 reset not still connected to 0x555ce28676c0 2024-08-31T22:36:16.247 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:16.245+0000 7f338ab71640 -1 reset not still connected to 0x555ce2867a00 2024-08-31T22:36:16.247 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:16.245+0000 7f338ab71640 -1 reset not still connected to 0x555ce58b2b60 2024-08-31T22:36:16.247 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:16.245+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:36:16.247 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:16.245+0000 7f338ab71640 -1 reset not still connected to 0x555cf25991e0 2024-08-31T22:36:16.247 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:16.245+0000 7f338ab71640 -1 reset not still connected to 0x555cf7f8f450 2024-08-31T22:36:16.247 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:16.245+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:36:16.248 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:16.245+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:36:16.248 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:16.245+0000 7f338ab71640 -1 reset not still connected to 0x555d0eeeaa90 2024-08-31T22:36:16.248 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:16.245+0000 7f338ab71640 -1 reset not still connected to 0x555d134d2000 2024-08-31T22:36:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 66790400 bytes 2024-08-31T22:36:18.507 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:18.509+0000 7f338ab71640 -1 reset not still connected to 0x555ce28676c0 2024-08-31T22:36:18.507 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:18.509+0000 7f338ab71640 -1 reset not still connected to 0x555ce2867a00 2024-08-31T22:36:18.507 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:18.509+0000 7f338ab71640 -1 reset not still connected to 0x555ce58b2b60 2024-08-31T22:36:18.507 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:18.509+0000 7f338ab71640 -1 reset not still connected to 0x555ce619a680 2024-08-31T22:36:18.507 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:18.509+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:36:18.507 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:18.509+0000 7f338ab71640 -1 reset not still connected to 0x555cf25991e0 2024-08-31T22:36:18.512 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:18.513+0000 7f338ab71640 -1 reset not still connected to 0x555cf7f8f450 2024-08-31T22:36:18.512 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:18.513+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:36:18.512 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:18.513+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:36:18.512 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:18.513+0000 7f338ab71640 -1 reset not still connected to 0x555d0eeeaa90 2024-08-31T22:36:18.512 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:18.513+0000 7f338ab71640 -1 reset not still connected to 0x555d134d2000 2024-08-31T22:36:18.562 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:18.557+0000 7f338ab71640 -1 reset not still connected to 0x555ce28676c0 2024-08-31T22:36:18.562 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:18.561+0000 7f338ab71640 -1 reset not still connected to 0x555ce2867a00 2024-08-31T22:36:18.563 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:18.561+0000 7f338ab71640 -1 reset not still connected to 0x555ce58b2b60 2024-08-31T22:36:18.563 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:18.561+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:36:18.563 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:18.561+0000 7f338ab71640 -1 reset not still connected to 0x555cf25991e0 2024-08-31T22:36:18.563 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:18.561+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:36:18.563 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:18.561+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:36:18.563 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:18.561+0000 7f338ab71640 -1 reset not still connected to 0x555d134d2000 2024-08-31T22:36:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 83017728 bytes 2024-08-31T22:36:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:36:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:36:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:36:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:36:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:36:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_27 2024-08-31T22:36:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 journal 128M 2024-08-31T22:36:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:36:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_27 2024-08-31T22:36:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:36:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:36:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_27 128M 2024-08-31T22:36:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:36:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_27 2024-08-31T22:36:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:36:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:36:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:36:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_27 2024-08-31T22:36:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:36:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:36:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:36:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:36:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:36:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_27 2024-08-31T22:36:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:36:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_27 2024-08-31T22:36:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:36:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=5 2024-08-31T22:36:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:36:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:36:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 65992704 bytes 2024-08-31T22:36:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:36:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:36:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:36:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:36:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:36:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_28 2024-08-31T22:36:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 journal 128M 2024-08-31T22:36:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:36:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_28 2024-08-31T22:36:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:36:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:36:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_28 128M 2024-08-31T22:36:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:36:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_28 2024-08-31T22:36:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:36:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:36:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:36:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_28 2024-08-31T22:36:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:36:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:36:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:36:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:36:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:36:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_28 2024-08-31T22:36:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:36:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_28 2024-08-31T22:36:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:36:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=10 2024-08-31T22:36:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:36:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:36:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 621568 bytes 2024-08-31T22:36:26.277 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:26.277+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:36:26.277 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:26.277+0000 7f338ab71640 -1 reset not still connected to 0x555cf7f8f450 2024-08-31T22:36:26.277 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:26.277+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:36:26.277 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:26.277+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:36:26.279 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:26.281+0000 7f338ab71640 -1 reset not still connected to 0x555d0eeeaa90 2024-08-31T22:36:26.448 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:26.449+0000 7f338ab71640 -1 reset not still connected to 0x555ce133fc70 2024-08-31T22:36:26.448 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:26.449+0000 7f338ab71640 -1 reset not still connected to 0x555ce133fe10 2024-08-31T22:36:26.449 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:26.449+0000 7f338ab71640 -1 reset not still connected to 0x555ce2638000 2024-08-31T22:36:26.449 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:26.449+0000 7f338ab71640 -1 reset not still connected to 0x555ce26391e0 2024-08-31T22:36:26.449 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:26.449+0000 7f338ab71640 -1 reset not still connected to 0x555ce2673450 2024-08-31T22:36:26.449 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:26.449+0000 7f338ab71640 -1 reset not still connected to 0x555ce28676c0 2024-08-31T22:36:26.449 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:26.449+0000 7f338ab71640 -1 reset not still connected to 0x555ce2867a00 2024-08-31T22:36:26.449 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:26.449+0000 7f338ab71640 -1 reset not still connected to 0x555ce58b2b60 2024-08-31T22:36:26.449 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:26.449+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:36:26.449 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:26.449+0000 7f338ab71640 -1 reset not still connected to 0x555cf25991e0 2024-08-31T22:36:26.449 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:26.449+0000 7f338ab71640 -1 reset not still connected to 0x555cf7f8f450 2024-08-31T22:36:26.449 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:26.449+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:36:26.449 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:26.449+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:36:26.449 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:26.449+0000 7f338ab71640 -1 reset not still connected to 0x555d0eeeaa90 2024-08-31T22:36:28.581 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:28.581+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:36:28.586 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:28.589+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:36:28.586 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:28.589+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:36:28.586 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:28.589+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:36:32.131 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:36:32.133+0000 7f1a77b11640 -1 reset not still connected to 0x5649e11c6000 2024-08-31T22:36:32.131 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:36:32.133+0000 7f1a77b11640 -1 reset not still connected to 0x5649f276dee0 2024-08-31T22:36:32.318 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:32.317+0000 7f338ab71640 -1 reset not still connected to 0x555ce66a0dd0 2024-08-31T22:36:32.318 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:32.317+0000 7f338ab71640 -1 reset not still connected to 0x555d0a5768f0 2024-08-31T22:36:33.309 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:33.309+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:36:33.311 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:33.309+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:36:33.311 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:33.309+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:36:34.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 143578112 bytes 2024-08-31T22:36:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:36:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:36:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:36:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:36:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:36:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_29 2024-08-31T22:36:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 journal 128M 2024-08-31T22:36:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:36:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_29 2024-08-31T22:36:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:36:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:36:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_29 128M 2024-08-31T22:36:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:36:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_29 2024-08-31T22:36:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:36:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:36:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:36:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_29 2024-08-31T22:36:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:36:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:36:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:36:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:36:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:36:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_29 2024-08-31T22:36:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:36:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_29 2024-08-31T22:36:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:36:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=13 2024-08-31T22:36:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:36:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:36:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 58292224 bytes 2024-08-31T22:36:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 84736000 bytes 2024-08-31T22:36:44.928 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:44.925+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:36:44.928 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:44.925+0000 7f6588f42640 -1 reset not still connected to 0x5620404e49c0 2024-08-31T22:36:44.928 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:44.925+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:36:44.928 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:44.925+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:36:44.929 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:44.925+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:36:44.929 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:44.925+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:36:44.929 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:44.925+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:36:44.929 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:36:44.925+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:36:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 136306688 bytes 2024-08-31T22:36:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:36:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:36:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:36:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:36:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:36:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_30 2024-08-31T22:36:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 journal 128M 2024-08-31T22:36:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:36:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_30 2024-08-31T22:36:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:36:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:36:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_30 128M 2024-08-31T22:36:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:36:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_30 2024-08-31T22:36:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:36:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:36:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:36:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:36:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_30 2024-08-31T22:36:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:36:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:36:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:36:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:36:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:36:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_30 2024-08-31T22:36:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:36:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:36:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_30 2024-08-31T22:36:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:36:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=14 2024-08-31T22:36:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:36:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:36:51.358 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:51.353+0000 7f338ab71640 -1 reset not still connected to 0x555ce28676c0 2024-08-31T22:36:51.359 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:51.361+0000 7f338ab71640 -1 reset not still connected to 0x555ce2867a00 2024-08-31T22:36:51.359 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:51.361+0000 7f338ab71640 -1 reset not still connected to 0x555ce58b2b60 2024-08-31T22:36:51.360 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:51.361+0000 7f338ab71640 -1 reset not still connected to 0x555ce619a680 2024-08-31T22:36:51.360 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:51.361+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:36:51.360 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:51.361+0000 7f338ab71640 -1 reset not still connected to 0x555cf25991e0 2024-08-31T22:36:51.360 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:51.361+0000 7f338ab71640 -1 reset not still connected to 0x555cf7f8f450 2024-08-31T22:36:51.360 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:51.361+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:36:51.360 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:51.361+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:36:51.360 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:51.361+0000 7f338ab71640 -1 reset not still connected to 0x555d0eeeaa90 2024-08-31T22:36:51.361 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:51.361+0000 7f338ab71640 -1 reset not still connected to 0x555d134d2000 2024-08-31T22:36:51.361 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:51.361+0000 7f338ab71640 -1 reset not still connected to 0x555d1cc7c750 2024-08-31T22:36:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 46141440 bytes 2024-08-31T22:36:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 60155904 bytes 2024-08-31T22:36:53.795 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:36:53.797+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:36:53.796 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:36:53.797+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:36:53.796 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:36:53.797+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:36:53.796 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:36:53.797+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:36:54.783 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:36:54.785+0000 7f1a77b11640 -1 reset not still connected to 0x5649c0c960d0 2024-08-31T22:36:54.783 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:36:54.785+0000 7f1a77b11640 -1 reset not still connected to 0x5649c0c96b60 2024-08-31T22:36:54.786 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:36:54.785+0000 7f1a77b11640 -1 reset not still connected to 0x5649d2b4e4e0 2024-08-31T22:36:54.786 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:36:54.785+0000 7f1a77b11640 -1 reset not still connected to 0x5649d7389c70 2024-08-31T22:36:54.786 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:36:54.785+0000 7f1a77b11640 -1 reset not still connected to 0x5649dca20680 2024-08-31T22:36:54.787 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:36:54.785+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfc38000 2024-08-31T22:36:54.787 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:36:54.785+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:36:54.787 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:36:54.785+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:36:54.787 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:36:54.785+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:36:54.787 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:36:54.785+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:36:56.233 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:56.232+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:36:56.237 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:56.236+0000 7f338ab71640 -1 reset not still connected to 0x555cf25991e0 2024-08-31T22:36:56.237 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:56.236+0000 7f338ab71640 -1 reset not still connected to 0x555cf7f8f450 2024-08-31T22:36:56.237 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:56.236+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:36:56.237 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:56.236+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:36:56.242 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:56.244+0000 7f338ab71640 -1 reset not still connected to 0x555d1cc7c750 2024-08-31T22:36:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 94144512 bytes 2024-08-31T22:36:58.996 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:58.996+0000 7f338ab71640 -1 reset not still connected to 0x555ce2638a90 2024-08-31T22:36:58.996 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:58.996+0000 7f338ab71640 -1 reset not still connected to 0x555ce26392b0 2024-08-31T22:36:58.996 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:58.996+0000 7f338ab71640 -1 reset not still connected to 0x555ce2866000 2024-08-31T22:36:58.996 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:58.996+0000 7f338ab71640 -1 reset not still connected to 0x555ce293a410 2024-08-31T22:36:58.997 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:58.996+0000 7f338ab71640 -1 reset not still connected to 0x555d0a5768f0 2024-08-31T22:36:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 155108352 bytes 2024-08-31T22:36:59.352 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:59.352+0000 7f338ab71640 -1 reset not still connected to 0x555ce28676c0 2024-08-31T22:36:59.353 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:59.352+0000 7f338ab71640 -1 reset not still connected to 0x555ce2867a00 2024-08-31T22:36:59.355 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:59.356+0000 7f338ab71640 -1 reset not still connected to 0x555ce58b2b60 2024-08-31T22:36:59.356 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:59.356+0000 7f338ab71640 -1 reset not still connected to 0x555ce619a680 2024-08-31T22:36:59.356 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:59.356+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:36:59.356 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:59.356+0000 7f338ab71640 -1 reset not still connected to 0x555cf25991e0 2024-08-31T22:36:59.356 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:59.356+0000 7f338ab71640 -1 reset not still connected to 0x555cf7f8f450 2024-08-31T22:36:59.356 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:59.356+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:36:59.356 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:59.356+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:36:59.357 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:59.356+0000 7f338ab71640 -1 reset not still connected to 0x555d0eeeaa90 2024-08-31T22:36:59.357 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:59.356+0000 7f338ab71640 -1 reset not still connected to 0x555d134d2000 2024-08-31T22:36:59.358 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:36:59.360+0000 7f338ab71640 -1 reset not still connected to 0x555d1cc7c750 2024-08-31T22:36:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:500 IOs, 158658560 bytes 2024-08-31T22:37:01.027 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:37:01.024+0000 7f338ab71640 -1 reset not still connected to 0x555ce133e820 2024-08-31T22:37:01.027 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:37:01.028+0000 7f338ab71640 -1 reset not still connected to 0x555ce133fa00 2024-08-31T22:37:02.131 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.128+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:37:02.447 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.448+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:37:02.447 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.448+0000 7f6588f42640 -1 reset not still connected to 0x562058e9be10 2024-08-31T22:37:02.447 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.448+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:37:02.614 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.612+0000 7f6588f42640 -1 reset not still connected to 0x56202461dd40 2024-08-31T22:37:02.614 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.612+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:37:02.614 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.612+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:37:02.614 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.612+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:37:02.620 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.616+0000 7f6588f42640 -1 reset not still connected to 0x5620324664e0 2024-08-31T22:37:02.620 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.620+0000 7f6588f42640 -1 reset not still connected to 0x562037ec7ee0 2024-08-31T22:37:02.620 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.620+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:37:02.620 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.620+0000 7f6588f42640 -1 reset not still connected to 0x5620404e49c0 2024-08-31T22:37:02.620 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.620+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:37:02.620 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.620+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:37:02.620 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.620+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:37:02.620 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.620+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:37:02.620 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.620+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:37:02.621 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.620+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:37:02.621 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.620+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:37:02.621 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.620+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:37:02.631 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:02.632+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:37:03.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:600 IOs, 214339584 bytes 2024-08-31T22:37:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:37:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:37:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:37:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:37:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:37:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_31 2024-08-31T22:37:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 journal 128M 2024-08-31T22:37:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:37:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_31 2024-08-31T22:37:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:37:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:37:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_31 128M 2024-08-31T22:37:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:37:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_31 2024-08-31T22:37:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:37:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:37:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:37:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_31 2024-08-31T22:37:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:37:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:37:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:37:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:37:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:37:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_31 2024-08-31T22:37:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:37:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_31 2024-08-31T22:37:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:37:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=10 2024-08-31T22:37:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:37:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:37:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 77815808 bytes 2024-08-31T22:37:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 78456832 bytes 2024-08-31T22:37:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 103252992 bytes 2024-08-31T22:37:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 154537984 bytes 2024-08-31T22:37:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:37:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:37:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:37:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:37:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:37:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_32 2024-08-31T22:37:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 journal 128M 2024-08-31T22:37:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:37:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_32 2024-08-31T22:37:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:37:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:37:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_32 128M 2024-08-31T22:37:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:37:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_32 2024-08-31T22:37:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:37:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:37:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:37:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_32 2024-08-31T22:37:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:37:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:37:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:37:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:37:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:37:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_32 2024-08-31T22:37:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:37:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_32 2024-08-31T22:37:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:37:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=6 2024-08-31T22:37:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:37:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:37:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 15667200 bytes 2024-08-31T22:37:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 19824640 bytes 2024-08-31T22:37:18.533 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:37:18.532+0000 7f13efcb9640 -1 reset not still connected to 0x559f1de35380 2024-08-31T22:37:18.533 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:37:18.532+0000 7f13efcb9640 -1 reset not still connected to 0x559f1e38d110 2024-08-31T22:37:18.533 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:37:18.532+0000 7f13efcb9640 -1 reset not still connected to 0x559f219a5d40 2024-08-31T22:37:18.535 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:37:18.536+0000 7f13efcb9640 -1 reset not still connected to 0x559f26ea96c0 2024-08-31T22:37:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 70220800 bytes 2024-08-31T22:37:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:37:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:37:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:37:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:37:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:37:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_33 2024-08-31T22:37:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 journal 128M 2024-08-31T22:37:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:37:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_33 2024-08-31T22:37:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:37:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:37:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_33 128M 2024-08-31T22:37:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:37:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_33 2024-08-31T22:37:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:37:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:37:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:37:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_33 2024-08-31T22:37:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:37:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:37:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:37:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:37:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:37:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_33 2024-08-31T22:37:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:37:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_33 2024-08-31T22:37:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:37:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=13 2024-08-31T22:37:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:37:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:37:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 36820992 bytes 2024-08-31T22:37:24.322 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:24.320+0000 7f6588f42640 -1 reset not still connected to 0x562032a40680 2024-08-31T22:37:24.322 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:24.320+0000 7f6588f42640 -1 reset not still connected to 0x562032a40d00 2024-08-31T22:37:24.322 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:24.320+0000 7f6588f42640 -1 reset not still connected to 0x562037235a00 2024-08-31T22:37:24.322 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:24.320+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:37:24.322 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:24.320+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:37:24.322 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:24.320+0000 7f6588f42640 -1 reset not still connected to 0x562045dee270 2024-08-31T22:37:24.322 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:24.320+0000 7f6588f42640 -1 reset not still connected to 0x56204cad00d0 2024-08-31T22:37:24.322 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:24.320+0000 7f6588f42640 -1 reset not still connected to 0x562054033ee0 2024-08-31T22:37:24.322 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:24.320+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:37:24.322 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:24.320+0000 7f6588f42640 -1 reset not still connected to 0x562058e9be10 2024-08-31T22:37:24.322 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:24.320+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:37:25.016 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:25.012+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:37:25.016 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:25.012+0000 7f6588f42640 -1 reset not still connected to 0x5620404e49c0 2024-08-31T22:37:25.016 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:25.012+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:37:25.016 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:25.012+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:37:25.016 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:25.012+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:37:25.018 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:25.016+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:37:25.018 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:25.016+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:37:25.019 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:25.016+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:37:25.020 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:25.020+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:37:25.020 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:25.020+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:37:25.020 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:25.020+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:37:25.020 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:25.020+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:37:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 85237760 bytes 2024-08-31T22:37:26.464 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:26.464+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:37:26.989 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:26.988+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:37:26.989 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:26.988+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:37:26.989 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:26.988+0000 7f6588f42640 -1 reset not still connected to 0x5620324664e0 2024-08-31T22:37:26.989 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:26.988+0000 7f6588f42640 -1 reset not still connected to 0x562037ec7ee0 2024-08-31T22:37:26.989 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:26.988+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:37:26.989 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:26.988+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:37:26.989 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:26.988+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:37:26.989 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:26.988+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:37:26.989 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:26.988+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:37:26.989 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:26.988+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:37:26.990 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:26.988+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:37:26.990 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:26.988+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:37:26.990 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:26.988+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:37:26.990 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:26.988+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:37:26.990 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:26.988+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:37:27.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 99083264 bytes 2024-08-31T22:37:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 137629696 bytes 2024-08-31T22:37:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:500 IOs, 159076352 bytes 2024-08-31T22:37:31.324 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:37:31.324+0000 7f13efcb9640 -1 reset not still connected to 0x559f1de35380 2024-08-31T22:37:31.324 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:37:31.324+0000 7f13efcb9640 -1 reset not still connected to 0x559f1e38d110 2024-08-31T22:37:31.324 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:37:31.324+0000 7f13efcb9640 -1 reset not still connected to 0x559f219a5d40 2024-08-31T22:37:32.241 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:32.240+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:37:32.242 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:32.240+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:37:32.242 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:32.240+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:37:32.242 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:32.240+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:37:32.242 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:32.240+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:37:32.242 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:32.240+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:37:32.242 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:32.240+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:37:32.242 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:32.240+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:37:32.242 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:32.240+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:37:32.242 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:32.240+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:37:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:37:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:37:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:37:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:37:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:37:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_34 2024-08-31T22:37:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 journal 128M 2024-08-31T22:37:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:37:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_34 2024-08-31T22:37:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:37:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:37:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_34 128M 2024-08-31T22:37:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:37:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_34 2024-08-31T22:37:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:37:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:37:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:37:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_34 2024-08-31T22:37:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:37:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:37:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:37:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:37:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:37:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_34 2024-08-31T22:37:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:37:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_34 2024-08-31T22:37:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:37:33.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=13 2024-08-31T22:37:33.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:37:33.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:37:34.894 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:37:34.896+0000 7fdf7b115640 -1 reset not still connected to 0x55ddfb08a0d0 2024-08-31T22:37:37.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 57776128 bytes 2024-08-31T22:37:37.354 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:37.352+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:37:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 139771904 bytes 2024-08-31T22:37:45.480 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:45.479+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:37:45.481 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:45.479+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:37:45.720 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:45.711+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:37:45.720 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:45.715+0000 7f6588f42640 -1 reset not still connected to 0x5620404e49c0 2024-08-31T22:37:45.721 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:45.719+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:37:45.721 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:45.719+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:37:45.721 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:45.719+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:37:45.721 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:45.719+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:37:45.721 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:45.719+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:37:45.722 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:45.719+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:37:45.722 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:45.719+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:37:45.722 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:45.719+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:37:45.722 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:45.719+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:37:45.722 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:45.719+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:37:46.255 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:46.247+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:37:46.255 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:46.247+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:37:46.255 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:46.247+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:37:46.255 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:46.247+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:37:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:37:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:37:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:37:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:37:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:37:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_35 2024-08-31T22:37:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 journal 128M 2024-08-31T22:37:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:37:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_35 2024-08-31T22:37:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:37:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:37:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_35 128M 2024-08-31T22:37:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:37:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_35 2024-08-31T22:37:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:37:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:37:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:37:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:37:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_35 2024-08-31T22:37:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:37:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:37:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:37:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:37:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:37:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_35 2024-08-31T22:37:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:37:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:37:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_35 2024-08-31T22:37:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:37:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=14 2024-08-31T22:37:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:37:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:37:47.928 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:47.927+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:37:47.929 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:47.927+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:37:47.929 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:47.927+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:37:47.929 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:47.927+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:37:47.929 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:47.927+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:37:47.929 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:47.927+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:37:47.929 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:37:47.927+0000 7f6588f42640 -1 reset not still connected to 0x56202461ca90 2024-08-31T22:37:49.121 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:37:49.119+0000 7f338ab71640 -1 reset not still connected to 0x555d168d1790 2024-08-31T22:37:52.295 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:37:52.291+0000 7f1a77b11640 -1 reset not still connected to 0x5649d7389c70 2024-08-31T22:37:52.295 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:37:52.291+0000 7f1a77b11640 -1 reset not still connected to 0x5649dca20680 2024-08-31T22:37:52.295 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:37:52.291+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfc38000 2024-08-31T22:37:52.295 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:37:52.291+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:37:52.301 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:37:52.299+0000 7f1a77b11640 -1 reset not still connected to 0x5649e235e680 2024-08-31T22:37:52.302 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:37:52.299+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:37:52.302 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:37:52.299+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:37:52.302 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:37:52.299+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:37:52.601 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:37:52.600+0000 7ff273ffe640 -1 reset not still connected to 0x5619387ba270 2024-08-31T22:37:52.601 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:37:52.600+0000 7ff273ffe640 -1 reset not still connected to 0x56194307e820 2024-08-31T22:37:52.601 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:37:52.600+0000 7ff273ffe640 -1 reset not still connected to 0x56194cb5b520 2024-08-31T22:37:52.601 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:37:52.600+0000 7ff273ffe640 -1 reset not still connected to 0x56194e533ad0 2024-08-31T22:37:52.601 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:37:52.604+0000 7ff273ffe640 -1 reset not still connected to 0x5619515f5110 2024-08-31T22:37:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 93621248 bytes 2024-08-31T22:37:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 115386368 bytes 2024-08-31T22:37:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 152152064 bytes 2024-08-31T22:38:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:38:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:38:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:38:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:38:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:38:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_36 2024-08-31T22:38:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 journal 128M 2024-08-31T22:38:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:38:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_36 2024-08-31T22:38:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:38:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:38:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_36 128M 2024-08-31T22:38:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:38:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_36 2024-08-31T22:38:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:38:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:38:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:38:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_36 2024-08-31T22:38:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:38:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:38:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:38:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:38:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:38:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_36 2024-08-31T22:38:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:38:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_36 2024-08-31T22:38:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:38:00.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=8 2024-08-31T22:38:00.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:38:00.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:38:01.143 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.143+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:38:01.143 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.143+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:38:01.152 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.151+0000 7f338ab71640 -1 reset not still connected to 0x555d134d2000 2024-08-31T22:38:01.152 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.151+0000 7f338ab71640 -1 reset not still connected to 0x555d1cc7c750 2024-08-31T22:38:01.155 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.151+0000 7f338ab71640 -1 reset not still connected to 0x555ce133fe10 2024-08-31T22:38:01.156 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.155+0000 7f338ab71640 -1 reset not still connected to 0x555ce26391e0 2024-08-31T22:38:01.162 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.159+0000 7f338ab71640 -1 reset not still connected to 0x555ce2673450 2024-08-31T22:38:01.162 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.159+0000 7f338ab71640 -1 reset not still connected to 0x555ce28676c0 2024-08-31T22:38:01.164 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.163+0000 7f338ab71640 -1 reset not still connected to 0x555ce2867a00 2024-08-31T22:38:01.164 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.163+0000 7f338ab71640 -1 reset not still connected to 0x555ce58b2b60 2024-08-31T22:38:01.164 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.163+0000 7f338ab71640 -1 reset not still connected to 0x555cee1ea8f0 2024-08-31T22:38:01.165 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.163+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:38:01.165 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.163+0000 7f338ab71640 -1 reset not still connected to 0x555cf25991e0 2024-08-31T22:38:01.165 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.163+0000 7f338ab71640 -1 reset not still connected to 0x555cf7f8f450 2024-08-31T22:38:01.165 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.163+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:38:01.165 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.163+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:38:01.165 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.163+0000 7f338ab71640 -1 reset not still connected to 0x555d0eeeaa90 2024-08-31T22:38:01.165 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.163+0000 7f338ab71640 -1 reset not still connected to 0x555d134d2000 2024-08-31T22:38:01.165 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.163+0000 7f338ab71640 -1 reset not still connected to 0x555d1cc7c750 2024-08-31T22:38:01.666 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:01.663+0000 7f338ab71640 -1 reset not still connected to 0x555d1cc7c750 2024-08-31T22:38:02.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 20469760 bytes 2024-08-31T22:38:02.985 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:02.983+0000 7f338ab71640 -1 reset not still connected to 0x555ce26391e0 2024-08-31T22:38:02.986 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:02.983+0000 7f338ab71640 -1 reset not still connected to 0x555ce2673450 2024-08-31T22:38:02.987 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:02.987+0000 7f338ab71640 -1 reset not still connected to 0x555ce2867a00 2024-08-31T22:38:02.987 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:02.987+0000 7f338ab71640 -1 reset not still connected to 0x555ce58b2b60 2024-08-31T22:38:02.987 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:02.987+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:38:02.990 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:02.987+0000 7f338ab71640 -1 reset not still connected to 0x555cf25991e0 2024-08-31T22:38:02.990 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:02.987+0000 7f338ab71640 -1 reset not still connected to 0x555cf7f8f450 2024-08-31T22:38:02.990 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:02.987+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:38:02.990 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:02.987+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:38:02.990 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:02.987+0000 7f338ab71640 -1 reset not still connected to 0x555d134d2000 2024-08-31T22:38:02.991 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:02.987+0000 7f338ab71640 -1 reset not still connected to 0x555d1cc7c750 2024-08-31T22:38:03.379 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:03.375+0000 7f1a77b11640 -1 reset not still connected to 0x5649c0752750 2024-08-31T22:38:03.389 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:03.387+0000 7f1a77b11640 -1 reset not still connected to 0x5649c0c96b60 2024-08-31T22:38:03.389 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:03.387+0000 7f1a77b11640 -1 reset not still connected to 0x5649d2b4e4e0 2024-08-31T22:38:03.389 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:03.387+0000 7f1a77b11640 -1 reset not still connected to 0x5649d3f1c340 2024-08-31T22:38:03.389 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:03.387+0000 7f1a77b11640 -1 reset not still connected to 0x5649d7389c70 2024-08-31T22:38:03.389 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:03.387+0000 7f1a77b11640 -1 reset not still connected to 0x5649dca20680 2024-08-31T22:38:03.390 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:03.387+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfc38000 2024-08-31T22:38:03.390 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:03.387+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:38:03.390 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:03.387+0000 7f1a77b11640 -1 reset not still connected to 0x5649e235e680 2024-08-31T22:38:03.390 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:03.387+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:38:03.390 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:03.387+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:38:03.390 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:03.387+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:38:08.406 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:08.403+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:38:08.407 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:08.403+0000 7f6588f42640 -1 reset not still connected to 0x56202611d040 2024-08-31T22:38:08.407 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:08.403+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:38:08.407 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:08.403+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:38:08.407 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:08.403+0000 7f6588f42640 -1 reset not still connected to 0x5620324664e0 2024-08-31T22:38:08.411 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:08.407+0000 7f6588f42640 -1 reset not still connected to 0x562037ec7ee0 2024-08-31T22:38:08.411 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:08.407+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:38:08.411 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:08.407+0000 7f6588f42640 -1 reset not still connected to 0x5620404e49c0 2024-08-31T22:38:08.412 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:08.407+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:38:08.412 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:08.407+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:38:08.412 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:08.407+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:38:08.412 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:08.407+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:38:08.412 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:08.407+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:38:08.412 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:08.407+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:38:08.412 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:08.407+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:38:08.412 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:08.407+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:38:08.412 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:08.407+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:38:08.412 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:08.407+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:38:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:38:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:38:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:38:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:38:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:38:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_37 2024-08-31T22:38:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 journal 128M 2024-08-31T22:38:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:38:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_37 2024-08-31T22:38:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:38:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:38:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_37 128M 2024-08-31T22:38:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:38:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_37 2024-08-31T22:38:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:38:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:38:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:38:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_37 2024-08-31T22:38:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:38:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:38:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:38:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:38:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:38:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_37 2024-08-31T22:38:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:38:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_37 2024-08-31T22:38:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:38:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=14 2024-08-31T22:38:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:38:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:38:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 41427968 bytes 2024-08-31T22:38:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 66128896 bytes 2024-08-31T22:38:14.872 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:14.871+0000 7f6588f42640 -1 reset not still connected to 0x562029759ba0 2024-08-31T22:38:14.872 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:14.871+0000 7f6588f42640 -1 reset not still connected to 0x56202997c9c0 2024-08-31T22:38:14.873 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:14.871+0000 7f6588f42640 -1 reset not still connected to 0x5620326af930 2024-08-31T22:38:14.873 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:14.871+0000 7f6588f42640 -1 reset not still connected to 0x562032a40680 2024-08-31T22:38:14.873 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:14.871+0000 7f6588f42640 -1 reset not still connected to 0x562032a40d00 2024-08-31T22:38:14.873 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:14.871+0000 7f6588f42640 -1 reset not still connected to 0x562037235a00 2024-08-31T22:38:14.873 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:14.871+0000 7f6588f42640 -1 reset not still connected to 0x562037adec30 2024-08-31T22:38:14.873 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:14.871+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:38:14.873 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:14.871+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:38:14.873 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:14.871+0000 7f6588f42640 -1 reset not still connected to 0x562045dee270 2024-08-31T22:38:14.873 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:14.871+0000 7f6588f42640 -1 reset not still connected to 0x56204cad00d0 2024-08-31T22:38:14.874 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:14.871+0000 7f6588f42640 -1 reset not still connected to 0x562054033ee0 2024-08-31T22:38:14.874 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:14.871+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:38:14.874 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:14.871+0000 7f6588f42640 -1 reset not still connected to 0x562058e9be10 2024-08-31T22:38:14.874 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:14.871+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:38:15.984 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:15.983+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:38:15.984 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:15.983+0000 7f6588f42640 -1 reset not still connected to 0x5620404e49c0 2024-08-31T22:38:15.988 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:15.987+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:38:15.988 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:15.987+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:38:15.988 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:15.987+0000 7f6588f42640 -1 reset not still connected to 0x562045a6a270 2024-08-31T22:38:15.988 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:15.987+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:38:15.988 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:15.987+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:38:15.988 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:15.987+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:38:15.988 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:15.987+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:38:15.988 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:15.987+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:38:15.988 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:15.987+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:38:15.988 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:15.987+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:38:15.989 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:15.987+0000 7f6588f42640 -1 reset not still connected to 0x562058e2bc70 2024-08-31T22:38:15.989 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:15.987+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:38:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 107305984 bytes 2024-08-31T22:38:17.705 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:17.703+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:38:17.707 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:17.707+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:38:17.707 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:17.707+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:38:17.710 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:17.707+0000 7f6588f42640 -1 reset not still connected to 0x562037ec7ee0 2024-08-31T22:38:17.715 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:17.711+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:38:17.715 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:17.711+0000 7f6588f42640 -1 reset not still connected to 0x5620404e49c0 2024-08-31T22:38:17.718 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:17.711+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:38:17.719 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:17.715+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:38:17.719 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:17.715+0000 7f6588f42640 -1 reset not still connected to 0x562045a6a270 2024-08-31T22:38:17.719 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:17.715+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:38:17.720 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:17.715+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:38:17.720 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:17.715+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:38:17.720 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:17.715+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:38:17.720 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:17.715+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:38:17.721 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:17.715+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:38:17.721 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:17.719+0000 7f6588f42640 -1 reset not still connected to 0x562058e2bc70 2024-08-31T22:38:17.724 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:17.719+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:38:18.114 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:18.111+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:38:18.121 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:18.119+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:38:18.121 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:18.119+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:38:18.123 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:18.123+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:38:18.123 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:18.123+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:38:18.125 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:18.123+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:38:18.125 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:18.123+0000 7f6588f42640 -1 reset not still connected to 0x562058e2bc70 2024-08-31T22:38:18.125 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:18.123+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:38:18.218 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:18.215+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:38:20.326 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:20.323+0000 7f1a77b11640 -1 reset not still connected to 0x5649e235e680 2024-08-31T22:38:20.326 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:20.323+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:38:20.326 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:20.323+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:38:20.326 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:20.323+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:38:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:38:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:38:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:38:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:38:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:38:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_38 2024-08-31T22:38:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 journal 128M 2024-08-31T22:38:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:38:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_38 2024-08-31T22:38:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:38:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:38:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_38 128M 2024-08-31T22:38:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:38:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_38 2024-08-31T22:38:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:38:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:38:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:38:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_38 2024-08-31T22:38:23.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:38:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:38:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:38:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:38:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:38:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_38 2024-08-31T22:38:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:38:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_38 2024-08-31T22:38:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:38:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=11 2024-08-31T22:38:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:38:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:38:24.719 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:24.719+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:38:24.719 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:24.719+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:38:24.719 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:24.719+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:38:24.719 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:24.719+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:38:24.760 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:24.755+0000 7f1a77b11640 -1 reset not still connected to 0x5649d2b4e4e0 2024-08-31T22:38:24.760 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:24.755+0000 7f1a77b11640 -1 reset not still connected to 0x5649dca20680 2024-08-31T22:38:24.761 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:24.755+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfc38000 2024-08-31T22:38:24.761 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:24.759+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:38:24.761 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:24.759+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:38:24.761 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:24.759+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:38:25.261 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:25.259+0000 7f338ab71640 -1 reset not still connected to 0x555cf25991e0 2024-08-31T22:38:25.261 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:25.259+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:38:25.262 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:25.259+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:38:25.265 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:25.263+0000 7f338ab71640 -1 reset not still connected to 0x555d0eeeaa90 2024-08-31T22:38:25.265 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:25.263+0000 7f338ab71640 -1 reset not still connected to 0x555d134d2000 2024-08-31T22:38:25.266 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:25.263+0000 7f338ab71640 -1 reset not still connected to 0x555d1cc7c750 2024-08-31T22:38:25.502 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:25.499+0000 7f338ab71640 -1 reset not still connected to 0x555ce2638a90 2024-08-31T22:38:25.508 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:25.507+0000 7f338ab71640 -1 reset not still connected to 0x555ce26392b0 2024-08-31T22:38:25.509 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:25.507+0000 7f338ab71640 -1 reset not still connected to 0x555ce2866000 2024-08-31T22:38:25.509 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:25.507+0000 7f338ab71640 -1 reset not still connected to 0x555ce293a410 2024-08-31T22:38:25.509 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:25.507+0000 7f338ab71640 -1 reset not still connected to 0x555ce66a0dd0 2024-08-31T22:38:25.509 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:25.507+0000 7f338ab71640 -1 reset not still connected to 0x555cea62d5f0 2024-08-31T22:38:25.509 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:25.507+0000 7f338ab71640 -1 reset not still connected to 0x555d0a5768f0 2024-08-31T22:38:25.509 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:25.507+0000 7f338ab71640 -1 reset not still connected to 0x555d168d1790 2024-08-31T22:38:26.429 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.427+0000 7f338ab71640 -1 reset not still connected to 0x555ce293a410 2024-08-31T22:38:26.432 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.431+0000 7f338ab71640 -1 reset not still connected to 0x555cea62d5f0 2024-08-31T22:38:26.432 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.431+0000 7f338ab71640 -1 reset not still connected to 0x555d0a5768f0 2024-08-31T22:38:26.436 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.435+0000 7f338ab71640 -1 reset not still connected to 0x555d168d1790 2024-08-31T22:38:26.798 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.791+0000 7f338ab71640 -1 reset not still connected to 0x555ce26391e0 2024-08-31T22:38:26.798 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.795+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:38:26.798 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.795+0000 7f338ab71640 -1 reset not still connected to 0x555cf25991e0 2024-08-31T22:38:26.804 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.803+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:38:26.804 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.803+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:38:26.820 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.811+0000 7f338ab71640 -1 reset not still connected to 0x555d0eeeaa90 2024-08-31T22:38:26.820 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.819+0000 7f338ab71640 -1 reset not still connected to 0x555d134d2000 2024-08-31T22:38:26.820 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.819+0000 7f338ab71640 -1 reset not still connected to 0x555d1cc7c750 2024-08-31T22:38:26.824 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.823+0000 7f338ab71640 -1 reset not still connected to 0x555ce133e820 2024-08-31T22:38:26.828 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.827+0000 7f338ab71640 -1 reset not still connected to 0x555ce133fa00 2024-08-31T22:38:26.831 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.831+0000 7f338ab71640 -1 reset not still connected to 0x555ce2638a90 2024-08-31T22:38:26.831 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.831+0000 7f338ab71640 -1 reset not still connected to 0x555ce26392b0 2024-08-31T22:38:26.831 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.831+0000 7f338ab71640 -1 reset not still connected to 0x555ce2866000 2024-08-31T22:38:26.831 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.831+0000 7f338ab71640 -1 reset not still connected to 0x555ce293a410 2024-08-31T22:38:26.832 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.831+0000 7f338ab71640 -1 reset not still connected to 0x555ce66a0dd0 2024-08-31T22:38:26.832 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.831+0000 7f338ab71640 -1 reset not still connected to 0x555cea62d5f0 2024-08-31T22:38:26.832 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.831+0000 7f338ab71640 -1 reset not still connected to 0x555d0a5768f0 2024-08-31T22:38:26.832 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:26.831+0000 7f338ab71640 -1 reset not still connected to 0x555d168d1790 2024-08-31T22:38:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 94304256 bytes 2024-08-31T22:38:30.584 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:38:30.583+0000 7fdf7b115640 -1 reset not still connected to 0x55dde2fe1a00 2024-08-31T22:38:30.584 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:38:30.583+0000 7fdf7b115640 -1 reset not still connected to 0x55dde4411450 2024-08-31T22:38:30.584 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:38:30.583+0000 7fdf7b115640 -1 reset not still connected to 0x55dde49f60d0 2024-08-31T22:38:30.584 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:38:30.583+0000 7fdf7b115640 -1 reset not still connected to 0x55dde53556c0 2024-08-31T22:38:33.204 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:33.203+0000 7f1a77b11640 -1 reset not still connected to 0x5649beff6270 2024-08-31T22:38:33.205 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:33.203+0000 7f1a77b11640 -1 reset not still connected to 0x5649e11c6000 2024-08-31T22:38:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 152368128 bytes 2024-08-31T22:38:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:38:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:38:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:38:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:38:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:38:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_39 2024-08-31T22:38:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 journal 128M 2024-08-31T22:38:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:38:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_39 2024-08-31T22:38:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:38:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:38:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_39 128M 2024-08-31T22:38:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:38:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_39 2024-08-31T22:38:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:38:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:38:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:38:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:34.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_39 2024-08-31T22:38:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:38:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:38:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:38:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:38:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:38:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_39 2024-08-31T22:38:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:38:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_39 2024-08-31T22:38:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:38:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=10 2024-08-31T22:38:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:38:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:38:34.661 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:34.659+0000 7f338ab71640 -1 reset not still connected to 0x555ce66a0dd0 2024-08-31T22:38:34.661 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:34.659+0000 7f338ab71640 -1 reset not still connected to 0x555cea62d5f0 2024-08-31T22:38:34.661 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:34.659+0000 7f338ab71640 -1 reset not still connected to 0x555d0a5768f0 2024-08-31T22:38:34.661 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:38:34.659+0000 7f338ab71640 -1 reset not still connected to 0x555d168d1790 2024-08-31T22:38:35.710 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:35.703+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:38:35.710 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:35.703+0000 7f6588f42640 -1 reset not still connected to 0x562058e9be10 2024-08-31T22:38:35.710 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:35.703+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:38:36.808 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x56202461ca90 2024-08-31T22:38:36.809 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x56202461dba0 2024-08-31T22:38:36.809 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x56202461dd40 2024-08-31T22:38:36.809 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x56202499a0d0 2024-08-31T22:38:36.809 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:38:36.809 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:38:36.809 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:38:36.809 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x5620324664e0 2024-08-31T22:38:36.809 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x562037ec7ee0 2024-08-31T22:38:36.810 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:38:36.810 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x5620404e49c0 2024-08-31T22:38:36.810 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:38:36.810 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:38:36.810 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x562045a6a270 2024-08-31T22:38:36.810 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:38:36.810 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:38:36.810 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:38:36.810 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:38:36.810 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:38:36.810 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:38:36.810 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:38:36.810 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x562058e2bc70 2024-08-31T22:38:36.811 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:36.802+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:38:38.004 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:38.002+0000 7f1a77b11640 -1 reset not still connected to 0x5649d2103ee0 2024-08-31T22:38:38.004 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:38.002+0000 7f1a77b11640 -1 reset not still connected to 0x5649dc12add0 2024-08-31T22:38:38.004 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:38.002+0000 7f1a77b11640 -1 reset not still connected to 0x5649dc64c270 2024-08-31T22:38:38.004 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:38.002+0000 7f1a77b11640 -1 reset not still connected to 0x5649e11c6000 2024-08-31T22:38:38.004 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:38.002+0000 7f1a77b11640 -1 reset not still connected to 0x5649f276dee0 2024-08-31T22:38:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 60983296 bytes 2024-08-31T22:38:39.776 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:39.774+0000 7f1a77b11640 -1 reset not still connected to 0x5649d2b4e4e0 2024-08-31T22:38:39.779 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:39.774+0000 7f1a77b11640 -1 reset not still connected to 0x5649d7389c70 2024-08-31T22:38:39.779 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:39.774+0000 7f1a77b11640 -1 reset not still connected to 0x5649dca20680 2024-08-31T22:38:39.779 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:39.774+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfc38000 2024-08-31T22:38:39.779 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:39.774+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:38:39.779 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:39.774+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:38:39.779 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:39.774+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:38:41.995 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:41.990+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:38:41.997 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:41.994+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:38:41.997 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:41.994+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:38:42.000 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:41.998+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:38:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 113860608 bytes 2024-08-31T22:38:43.560 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:43.558+0000 7f1a77b11640 -1 reset not still connected to 0x5649c0752750 2024-08-31T22:38:43.562 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:43.558+0000 7f1a77b11640 -1 reset not still connected to 0x5649c0c960d0 2024-08-31T22:38:43.562 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:43.558+0000 7f1a77b11640 -1 reset not still connected to 0x5649c0c96b60 2024-08-31T22:38:43.562 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:43.558+0000 7f1a77b11640 -1 reset not still connected to 0x5649d2b4e4e0 2024-08-31T22:38:43.564 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:43.558+0000 7f1a77b11640 -1 reset not still connected to 0x5649d3f1c340 2024-08-31T22:38:43.564 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:43.558+0000 7f1a77b11640 -1 reset not still connected to 0x5649d7389c70 2024-08-31T22:38:43.564 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:43.558+0000 7f1a77b11640 -1 reset not still connected to 0x5649dca20680 2024-08-31T22:38:43.564 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:43.558+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfc38000 2024-08-31T22:38:43.564 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:43.558+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:38:43.564 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:43.558+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:38:43.564 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:43.558+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:38:43.564 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:43.558+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:38:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:38:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:38:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:38:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:38:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:38:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_40 2024-08-31T22:38:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 journal 128M 2024-08-31T22:38:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:38:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_40 2024-08-31T22:38:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:38:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:38:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_40 128M 2024-08-31T22:38:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:38:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_40 2024-08-31T22:38:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:38:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:38:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:38:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_40 2024-08-31T22:38:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:38:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:38:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:38:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:38:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:38:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_40 2024-08-31T22:38:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:38:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_40 2024-08-31T22:38:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:38:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=13 2024-08-31T22:38:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:38:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:38:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 2753536 bytes 2024-08-31T22:38:44.943 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:44.934+0000 7f1a77b11640 -1 reset not still connected to 0x5649c0c960d0 2024-08-31T22:38:44.943 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:44.934+0000 7f1a77b11640 -1 reset not still connected to 0x5649c0c96b60 2024-08-31T22:38:44.943 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:44.938+0000 7f1a77b11640 -1 reset not still connected to 0x5649d2b4e4e0 2024-08-31T22:38:44.943 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:44.938+0000 7f1a77b11640 -1 reset not still connected to 0x5649d7389c70 2024-08-31T22:38:44.943 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:44.938+0000 7f1a77b11640 -1 reset not still connected to 0x5649dca20680 2024-08-31T22:38:44.943 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:44.938+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfc38000 2024-08-31T22:38:44.943 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:44.938+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:38:44.943 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:44.938+0000 7f1a77b11640 -1 reset not still connected to 0x5649e235e680 2024-08-31T22:38:44.943 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:44.938+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:38:44.943 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:44.938+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:38:44.943 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:44.938+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:38:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 35622912 bytes 2024-08-31T22:38:46.537 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:46.534+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:38:46.537 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:46.534+0000 7f6588f42640 -1 reset not still connected to 0x562045dee270 2024-08-31T22:38:46.537 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:46.534+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:38:46.537 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:46.534+0000 7f6588f42640 -1 reset not still connected to 0x562058e9be10 2024-08-31T22:38:46.538 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:46.534+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:38:48.588 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:48.586+0000 7f1a77b11640 -1 reset not still connected to 0x5649dc12add0 2024-08-31T22:38:48.589 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:48.586+0000 7f1a77b11640 -1 reset not still connected to 0x5649dd3d6a90 2024-08-31T22:38:48.590 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:48.586+0000 7f1a77b11640 -1 reset not still connected to 0x5649e11c6000 2024-08-31T22:38:48.590 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:48.586+0000 7f1a77b11640 -1 reset not still connected to 0x5649f276dee0 2024-08-31T22:38:49.077 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.074+0000 7f1a77b11640 -1 reset not still connected to 0x5649bfb08750 2024-08-31T22:38:49.077 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.074+0000 7f1a77b11640 -1 reset not still connected to 0x5649c0752750 2024-08-31T22:38:49.077 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.074+0000 7f1a77b11640 -1 reset not still connected to 0x5649c0c960d0 2024-08-31T22:38:49.077 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.074+0000 7f1a77b11640 -1 reset not still connected to 0x5649c0c96b60 2024-08-31T22:38:49.077 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.074+0000 7f1a77b11640 -1 reset not still connected to 0x5649d2b4e4e0 2024-08-31T22:38:49.077 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.074+0000 7f1a77b11640 -1 reset not still connected to 0x5649d3f1c340 2024-08-31T22:38:49.078 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.074+0000 7f1a77b11640 -1 reset not still connected to 0x5649d7389c70 2024-08-31T22:38:49.078 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.074+0000 7f1a77b11640 -1 reset not still connected to 0x5649dca20680 2024-08-31T22:38:49.078 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.074+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfc38000 2024-08-31T22:38:49.078 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.074+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:38:49.078 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.074+0000 7f1a77b11640 -1 reset not still connected to 0x5649e235e680 2024-08-31T22:38:49.078 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.074+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:38:49.078 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.074+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:38:49.078 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.074+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:38:49.554 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.550+0000 7f1a77b11640 -1 reset not still connected to 0x5649beff6270 2024-08-31T22:38:49.560 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.554+0000 7f1a77b11640 -1 reset not still connected to 0x5649dc12add0 2024-08-31T22:38:49.560 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.554+0000 7f1a77b11640 -1 reset not still connected to 0x5649dc64c270 2024-08-31T22:38:49.561 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.554+0000 7f1a77b11640 -1 reset not still connected to 0x5649dd3d6a90 2024-08-31T22:38:49.561 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.554+0000 7f1a77b11640 -1 reset not still connected to 0x5649e11c6000 2024-08-31T22:38:49.565 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:49.554+0000 7f1a77b11640 -1 reset not still connected to 0x5649f276dee0 2024-08-31T22:38:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 69426176 bytes 2024-08-31T22:38:50.817 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:50.810+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:38:50.823 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:50.818+0000 7f6588f42640 -1 reset not still connected to 0x562046c83860 2024-08-31T22:38:50.823 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:50.818+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:38:50.823 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:50.822+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:38:50.826 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:50.822+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:38:50.826 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:50.822+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:38:50.826 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:50.822+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:38:50.826 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:50.822+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:38:50.826 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:50.822+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:38:50.826 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:50.822+0000 7f6588f42640 -1 reset not still connected to 0x56202461ca90 2024-08-31T22:38:50.826 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:50.822+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:38:50.826 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:50.822+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:38:50.826 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:50.822+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:38:50.826 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:50.822+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:38:51.092 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:51.090+0000 7f6588f42640 -1 reset not still connected to 0x56202997c9c0 2024-08-31T22:38:51.100 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:51.090+0000 7f6588f42640 -1 reset not still connected to 0x5620326af930 2024-08-31T22:38:51.100 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:51.090+0000 7f6588f42640 -1 reset not still connected to 0x562032a40680 2024-08-31T22:38:51.100 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:51.090+0000 7f6588f42640 -1 reset not still connected to 0x562032a40d00 2024-08-31T22:38:51.100 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:51.090+0000 7f6588f42640 -1 reset not still connected to 0x562037235a00 2024-08-31T22:38:51.100 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:51.090+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:38:51.101 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:51.094+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:38:51.101 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:51.094+0000 7f6588f42640 -1 reset not still connected to 0x562045dee270 2024-08-31T22:38:51.101 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:51.094+0000 7f6588f42640 -1 reset not still connected to 0x56204cad00d0 2024-08-31T22:38:51.101 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:51.094+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:38:51.101 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:51.094+0000 7f6588f42640 -1 reset not still connected to 0x562057bb5040 2024-08-31T22:38:51.101 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:51.094+0000 7f6588f42640 -1 reset not still connected to 0x562058e9be10 2024-08-31T22:38:51.101 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:51.094+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:38:52.150 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:52.146+0000 7f6588f42640 -1 reset not still connected to 0x56202997c9c0 2024-08-31T22:38:52.150 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:52.146+0000 7f6588f42640 -1 reset not still connected to 0x5620326af930 2024-08-31T22:38:52.150 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:52.146+0000 7f6588f42640 -1 reset not still connected to 0x562032a40680 2024-08-31T22:38:52.157 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:52.154+0000 7f6588f42640 -1 reset not still connected to 0x562032a40d00 2024-08-31T22:38:52.157 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:52.154+0000 7f6588f42640 -1 reset not still connected to 0x562036482000 2024-08-31T22:38:52.157 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:52.154+0000 7f6588f42640 -1 reset not still connected to 0x562037235a00 2024-08-31T22:38:52.157 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:52.154+0000 7f6588f42640 -1 reset not still connected to 0x562037adec30 2024-08-31T22:38:52.157 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:52.154+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:38:52.158 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:52.154+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:38:52.158 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:52.154+0000 7f6588f42640 -1 reset not still connected to 0x562045dee270 2024-08-31T22:38:52.158 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:52.154+0000 7f6588f42640 -1 reset not still connected to 0x56204cad00d0 2024-08-31T22:38:52.158 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:52.154+0000 7f6588f42640 -1 reset not still connected to 0x562054033ee0 2024-08-31T22:38:52.158 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:52.154+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:38:52.158 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:52.154+0000 7f6588f42640 -1 reset not still connected to 0x562057bb5040 2024-08-31T22:38:52.158 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:52.154+0000 7f6588f42640 -1 reset not still connected to 0x562058e9be10 2024-08-31T22:38:52.158 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:52.154+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:38:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 109659136 bytes 2024-08-31T22:38:53.259 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:53.250+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:38:55.899 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:38:55.898+0000 7fdf7b115640 -1 reset not still connected to 0x55ddd6dcfe10 2024-08-31T22:38:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:500 IOs, 166747136 bytes 2024-08-31T22:38:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:38:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:38:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:38:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:38:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:38:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_41 2024-08-31T22:38:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 journal 128M 2024-08-31T22:38:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:38:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_41 2024-08-31T22:38:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:38:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:38:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_41 128M 2024-08-31T22:38:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:38:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_41 2024-08-31T22:38:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:38:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:38:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:38:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:38:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_41 2024-08-31T22:38:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:38:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:38:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:38:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:38:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:38:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_41 2024-08-31T22:38:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:38:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:38:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_41 2024-08-31T22:38:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:38:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=13 2024-08-31T22:38:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:38:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:38:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 15122432 bytes 2024-08-31T22:38:59.408 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:59.406+0000 7f6588f42640 -1 reset not still connected to 0x5620404e49c0 2024-08-31T22:38:59.409 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:59.406+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:38:59.412 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:59.410+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:38:59.418 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:59.410+0000 7f6588f42640 -1 reset not still connected to 0x562045a6a270 2024-08-31T22:38:59.418 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:59.410+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:38:59.419 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:59.410+0000 7f6588f42640 -1 reset not still connected to 0x562046c83860 2024-08-31T22:38:59.419 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:59.410+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:38:59.419 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:59.410+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:38:59.419 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:59.410+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:38:59.419 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:59.410+0000 7f6588f42640 -1 reset not still connected to 0x562051276340 2024-08-31T22:38:59.419 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:59.410+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:38:59.419 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:59.410+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:38:59.419 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:59.410+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:38:59.420 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:38:59.418+0000 7f6588f42640 -1 reset not still connected to 0x56205d712b60 2024-08-31T22:38:59.664 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:38:59.662+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:39:00.553 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:00.550+0000 7f6588f42640 -1 reset not still connected to 0x562037235a00 2024-08-31T22:39:00.557 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:00.554+0000 7f6588f42640 -1 reset not still connected to 0x562037adec30 2024-08-31T22:39:00.560 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:00.558+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:39:00.560 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:00.558+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:39:00.560 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:00.558+0000 7f6588f42640 -1 reset not still connected to 0x562045dee270 2024-08-31T22:39:00.560 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:00.558+0000 7f6588f42640 -1 reset not still connected to 0x56204cad00d0 2024-08-31T22:39:00.562 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:00.558+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:39:00.562 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:00.558+0000 7f6588f42640 -1 reset not still connected to 0x562057bb5040 2024-08-31T22:39:00.563 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:00.558+0000 7f6588f42640 -1 reset not still connected to 0x562058e9be10 2024-08-31T22:39:00.563 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:00.558+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:39:00.674 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:00.670+0000 7f1a77b11640 -1 reset not still connected to 0x5649d2103ee0 2024-08-31T22:39:00.674 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:00.670+0000 7f1a77b11640 -1 reset not still connected to 0x5649dc12add0 2024-08-31T22:39:00.674 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:00.670+0000 7f1a77b11640 -1 reset not still connected to 0x5649dc64c270 2024-08-31T22:39:00.674 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:00.670+0000 7f1a77b11640 -1 reset not still connected to 0x5649dd3d6a90 2024-08-31T22:39:00.674 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:00.670+0000 7f1a77b11640 -1 reset not still connected to 0x5649e11c6000 2024-08-31T22:39:00.674 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:00.670+0000 7f1a77b11640 -1 reset not still connected to 0x5649f276dee0 2024-08-31T22:39:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 40436736 bytes 2024-08-31T22:39:01.022 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:01.014+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:39:01.022 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:01.014+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:39:02.061 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:02.058+0000 7f1a77b11640 -1 reset not still connected to 0x5649d2b4e4e0 2024-08-31T22:39:02.061 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:02.058+0000 7f1a77b11640 -1 reset not still connected to 0x5649d3f1c340 2024-08-31T22:39:02.061 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:02.058+0000 7f1a77b11640 -1 reset not still connected to 0x5649d7389c70 2024-08-31T22:39:02.061 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:02.058+0000 7f1a77b11640 -1 reset not still connected to 0x5649dca20680 2024-08-31T22:39:02.062 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:02.058+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfc38000 2024-08-31T22:39:02.065 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:02.062+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:39:02.067 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:02.066+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:39:02.067 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:02.066+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:39:02.074 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:02.070+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:39:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 117937152 bytes 2024-08-31T22:39:06.996 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:06.990+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:39:06.996 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:06.990+0000 7f338ab71640 -1 reset not still connected to 0x555d0eeeaa90 2024-08-31T22:39:06.997 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:06.994+0000 7f338ab71640 -1 reset not still connected to 0x555d1cc7c750 2024-08-31T22:39:07.094 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:07.090+0000 7f1a77b11640 -1 reset not still connected to 0x5649bba41930 2024-08-31T22:39:07.095 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:07.090+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfc38000 2024-08-31T22:39:07.095 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:07.090+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:39:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:39:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:39:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:39:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:39:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:39:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_42 2024-08-31T22:39:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 journal 128M 2024-08-31T22:39:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:39:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_42 2024-08-31T22:39:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:39:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:39:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_42 128M 2024-08-31T22:39:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:39:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_42 2024-08-31T22:39:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:39:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:39:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:39:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_42 2024-08-31T22:39:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:39:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:39:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:39:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:39:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:39:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_42 2024-08-31T22:39:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:39:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_42 2024-08-31T22:39:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:39:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=12 2024-08-31T22:39:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:39:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:39:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 29959168 bytes 2024-08-31T22:39:13.056 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:13.050+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:39:13.059 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:13.050+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:39:13.059 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:13.050+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:39:13.059 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:13.050+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:39:13.060 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:13.050+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:39:13.060 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:13.050+0000 7f6588f42640 -1 reset not still connected to 0x562058e2bc70 2024-08-31T22:39:13.060 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:13.050+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:39:13.060 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:13.050+0000 7f6588f42640 -1 reset not still connected to 0x56205d712b60 2024-08-31T22:39:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 74935296 bytes 2024-08-31T22:39:17.485 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:17.482+0000 7f338ab71640 -1 reset not still connected to 0x555ce26391e0 2024-08-31T22:39:17.486 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:17.482+0000 7f338ab71640 -1 reset not still connected to 0x555ce2673450 2024-08-31T22:39:17.491 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:17.486+0000 7f338ab71640 -1 reset not still connected to 0x555ce28676c0 2024-08-31T22:39:17.492 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:17.490+0000 7f338ab71640 -1 reset not still connected to 0x555ce2867a00 2024-08-31T22:39:17.492 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:17.490+0000 7f338ab71640 -1 reset not still connected to 0x555ce58b2b60 2024-08-31T22:39:17.495 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:17.494+0000 7f338ab71640 -1 reset not still connected to 0x555ce619a680 2024-08-31T22:39:17.495 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:17.494+0000 7f338ab71640 -1 reset not still connected to 0x555cee1ea8f0 2024-08-31T22:39:17.495 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:17.494+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:39:17.495 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:17.494+0000 7f338ab71640 -1 reset not still connected to 0x555cf25991e0 2024-08-31T22:39:17.495 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:17.494+0000 7f338ab71640 -1 reset not still connected to 0x555cf7f8f450 2024-08-31T22:39:17.495 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:17.494+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:39:17.495 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:17.494+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:39:17.495 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:17.494+0000 7f338ab71640 -1 reset not still connected to 0x555d0eeeaa90 2024-08-31T22:39:17.496 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:17.494+0000 7f338ab71640 -1 reset not still connected to 0x555d1114e8f0 2024-08-31T22:39:17.496 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:17.494+0000 7f338ab71640 -1 reset not still connected to 0x555d134d2000 2024-08-31T22:39:17.496 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:17.494+0000 7f338ab71640 -1 reset not still connected to 0x555d1c89bd40 2024-08-31T22:39:17.496 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:17.494+0000 7f338ab71640 -1 reset not still connected to 0x555d1cc7c750 2024-08-31T22:39:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 122971136 bytes 2024-08-31T22:39:20.181 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x5620324664e0 2024-08-31T22:39:20.181 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x562037ec7ee0 2024-08-31T22:39:20.181 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:39:20.181 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x5620404e49c0 2024-08-31T22:39:20.181 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:39:20.181 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:39:20.181 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x562045a6a270 2024-08-31T22:39:20.181 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:39:20.182 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x562046c82ea0 2024-08-31T22:39:20.182 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x562046c83860 2024-08-31T22:39:20.182 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:39:20.182 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:39:20.182 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:39:20.182 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x562051276340 2024-08-31T22:39:20.182 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:39:20.182 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:39:20.182 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:39:20.182 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x562058e2bc70 2024-08-31T22:39:20.182 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:39:20.182 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:20.178+0000 7f6588f42640 -1 reset not still connected to 0x56205d712b60 2024-08-31T22:39:21.330 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:21.326+0000 7f338ab71640 -1 reset not still connected to 0x555ce133e820 2024-08-31T22:39:21.330 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:21.326+0000 7f338ab71640 -1 reset not still connected to 0x555ce133fa00 2024-08-31T22:39:21.331 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:21.326+0000 7f338ab71640 -1 reset not still connected to 0x555ce162d110 2024-08-31T22:39:21.331 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:21.326+0000 7f338ab71640 -1 reset not still connected to 0x555ce2638a90 2024-08-31T22:39:21.331 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:21.326+0000 7f338ab71640 -1 reset not still connected to 0x555ce26392b0 2024-08-31T22:39:21.331 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:21.326+0000 7f338ab71640 -1 reset not still connected to 0x555ce2866000 2024-08-31T22:39:21.334 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:21.330+0000 7f338ab71640 -1 reset not still connected to 0x555d0a5768f0 2024-08-31T22:39:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:39:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:39:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:39:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:39:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:39:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_43 2024-08-31T22:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 journal 128M 2024-08-31T22:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_43 2024-08-31T22:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_43 128M 2024-08-31T22:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:39:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_43 2024-08-31T22:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_43 2024-08-31T22:39:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:39:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:39:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:39:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:39:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:39:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_43 2024-08-31T22:39:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:39:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_43 2024-08-31T22:39:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:39:22.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=9 2024-08-31T22:39:22.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:39:22.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:39:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 32026624 bytes 2024-08-31T22:39:26.120 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:26.122+0000 7f13efcb9640 -1 reset not still connected to 0x559f14700b60 2024-08-31T22:39:26.121 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:26.122+0000 7f13efcb9640 -1 reset not still connected to 0x559f19a8ed00 2024-08-31T22:39:26.121 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:26.122+0000 7f13efcb9640 -1 reset not still connected to 0x559f21abe410 2024-08-31T22:39:26.128 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:26.130+0000 7f13efcb9640 -1 reset not still connected to 0x559f2246c750 2024-08-31T22:39:26.128 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:26.130+0000 7f13efcb9640 -1 reset not still connected to 0x559f24292410 2024-08-31T22:39:26.128 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:26.130+0000 7f13efcb9640 -1 reset not still connected to 0x559f273195f0 2024-08-31T22:39:26.128 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:26.130+0000 7f13efcb9640 -1 reset not still connected to 0x559f287c15f0 2024-08-31T22:39:26.128 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:26.130+0000 7f13efcb9640 -1 reset not still connected to 0x559f2b7404e0 2024-08-31T22:39:26.128 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:26.130+0000 7f13efcb9640 -1 reset not still connected to 0x559f2d6d9c70 2024-08-31T22:39:26.129 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:26.130+0000 7f13efcb9640 -1 reset not still connected to 0x559f2e9f08f0 2024-08-31T22:39:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 61839360 bytes 2024-08-31T22:39:28.185 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:28.182+0000 7f6588f42640 -1 reset not still connected to 0x56202997c9c0 2024-08-31T22:39:28.186 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:28.182+0000 7f6588f42640 -1 reset not still connected to 0x5620326af930 2024-08-31T22:39:28.186 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:28.182+0000 7f6588f42640 -1 reset not still connected to 0x562032a40d00 2024-08-31T22:39:28.193 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:28.190+0000 7f6588f42640 -1 reset not still connected to 0x562037235a00 2024-08-31T22:39:28.193 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:28.190+0000 7f6588f42640 -1 reset not still connected to 0x562037adec30 2024-08-31T22:39:28.193 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:28.190+0000 7f6588f42640 -1 reset not still connected to 0x562038de0a90 2024-08-31T22:39:28.193 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:28.190+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:39:28.195 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:28.190+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:39:28.195 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:28.190+0000 7f6588f42640 -1 reset not still connected to 0x562045dee270 2024-08-31T22:39:28.198 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:28.194+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:39:28.198 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:28.194+0000 7f6588f42640 -1 reset not still connected to 0x562057bb5040 2024-08-31T22:39:28.198 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:28.194+0000 7f6588f42640 -1 reset not still connected to 0x562058e9be10 2024-08-31T22:39:28.198 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:28.194+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:39:29.705 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:29.702+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:39:29.705 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:29.702+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:39:29.705 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:29.702+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:39:29.705 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:29.706+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:39:29.705 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:29.706+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:39:29.727 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:29.722+0000 7f6588f42640 -1 reset not still connected to 0x562046c83860 2024-08-31T22:39:29.727 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:29.722+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:39:29.727 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:29.722+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:39:29.727 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:29.722+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:39:29.727 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:29.722+0000 7f6588f42640 -1 reset not still connected to 0x562051276340 2024-08-31T22:39:29.727 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:29.722+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:39:29.727 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:29.722+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:39:29.728 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:29.722+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:39:29.728 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:29.722+0000 7f6588f42640 -1 reset not still connected to 0x562058e2bc70 2024-08-31T22:39:29.728 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:29.722+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:39:29.728 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:29.722+0000 7f6588f42640 -1 reset not still connected to 0x56205d712b60 2024-08-31T22:39:30.579 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:30.582+0000 7f6588f42640 -1 reset not still connected to 0x56202997c9c0 2024-08-31T22:39:30.579 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:30.582+0000 7f6588f42640 -1 reset not still connected to 0x5620326af930 2024-08-31T22:39:30.583 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:30.582+0000 7f6588f42640 -1 reset not still connected to 0x562032a40680 2024-08-31T22:39:30.583 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:30.582+0000 7f6588f42640 -1 reset not still connected to 0x562032a40d00 2024-08-31T22:39:30.583 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:30.582+0000 7f6588f42640 -1 reset not still connected to 0x562037235a00 2024-08-31T22:39:30.583 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:30.582+0000 7f6588f42640 -1 reset not still connected to 0x562037adec30 2024-08-31T22:39:30.583 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:30.582+0000 7f6588f42640 -1 reset not still connected to 0x562038de0a90 2024-08-31T22:39:30.583 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:30.582+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:39:30.584 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:30.582+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:39:30.585 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:30.586+0000 7f6588f42640 -1 reset not still connected to 0x562057bb5040 2024-08-31T22:39:30.585 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:30.586+0000 7f6588f42640 -1 reset not still connected to 0x562058e9be10 2024-08-31T22:39:30.585 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:30.586+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:39:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:39:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:39:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:39:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:39:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:39:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_44 2024-08-31T22:39:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 journal 128M 2024-08-31T22:39:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:39:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_44 2024-08-31T22:39:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:39:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:39:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_44 128M 2024-08-31T22:39:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:39:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_44 2024-08-31T22:39:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:39:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:39:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:39:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_44 2024-08-31T22:39:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:39:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:31.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:39:31.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:39:31.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:39:31.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:39:31.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_44 2024-08-31T22:39:31.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:39:31.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:31.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_44 2024-08-31T22:39:31.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:39:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=12 2024-08-31T22:39:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:39:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:39:33.068 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.070+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:39:33.068 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.070+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:39:33.068 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.070+0000 7f6588f42640 -1 reset not still connected to 0x562037ec7ee0 2024-08-31T22:39:33.070 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.070+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:39:33.070 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.070+0000 7f6588f42640 -1 reset not still connected to 0x5620404e49c0 2024-08-31T22:39:33.070 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.070+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:39:33.073 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.074+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:39:33.073 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.074+0000 7f6588f42640 -1 reset not still connected to 0x562045a6a270 2024-08-31T22:39:33.073 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.074+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:39:33.073 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.074+0000 7f6588f42640 -1 reset not still connected to 0x562046a39380 2024-08-31T22:39:33.073 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.074+0000 7f6588f42640 -1 reset not still connected to 0x562046c83860 2024-08-31T22:39:33.073 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.074+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:39:33.074 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.074+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:39:33.074 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.074+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:39:33.074 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.074+0000 7f6588f42640 -1 reset not still connected to 0x562051276340 2024-08-31T22:39:33.074 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.074+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:39:33.074 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.074+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:39:33.074 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.074+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:39:33.074 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.074+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:39:33.074 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.074+0000 7f6588f42640 -1 reset not still connected to 0x56205d712b60 2024-08-31T22:39:33.375 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.373+0000 7f6588f42640 -1 reset not still connected to 0x56202461dd40 2024-08-31T22:39:33.375 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.373+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:39:33.375 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.373+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:39:33.375 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.373+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:39:33.376 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.373+0000 7f6588f42640 -1 reset not still connected to 0x562037ec7ee0 2024-08-31T22:39:33.380 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.381+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:39:33.380 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.381+0000 7f6588f42640 -1 reset not still connected to 0x5620404e49c0 2024-08-31T22:39:33.380 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.381+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:39:33.380 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.381+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:39:33.381 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.381+0000 7f6588f42640 -1 reset not still connected to 0x562045a6a270 2024-08-31T22:39:33.381 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.381+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:39:33.381 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.381+0000 7f6588f42640 -1 reset not still connected to 0x562046a39380 2024-08-31T22:39:33.381 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.381+0000 7f6588f42640 -1 reset not still connected to 0x562046c83860 2024-08-31T22:39:33.381 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.381+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:39:33.381 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.381+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:39:33.381 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.381+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:39:33.381 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.381+0000 7f6588f42640 -1 reset not still connected to 0x562051276340 2024-08-31T22:39:33.381 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.381+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:39:33.381 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.381+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:39:33.381 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.381+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:39:33.381 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:33.381+0000 7f6588f42640 -1 reset not still connected to 0x56205d712b60 2024-08-31T22:39:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 32909312 bytes 2024-08-31T22:39:34.540 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:34.537+0000 7f1a77b11640 -1 reset not still connected to 0x5649d2b4e4e0 2024-08-31T22:39:34.541 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:34.541+0000 7f1a77b11640 -1 reset not still connected to 0x5649dca20680 2024-08-31T22:39:34.542 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:34.541+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfc38000 2024-08-31T22:39:34.542 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:34.541+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:39:34.544 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:34.545+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:39:34.544 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:34.545+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:39:34.905 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:34.905+0000 7f1a77b11640 -1 reset not still connected to 0x5649caa220d0 2024-08-31T22:39:34.905 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:34.905+0000 7f1a77b11640 -1 reset not still connected to 0x5649d2b4e4e0 2024-08-31T22:39:34.905 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:34.905+0000 7f1a77b11640 -1 reset not still connected to 0x5649d3f1c340 2024-08-31T22:39:34.908 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:34.909+0000 7f1a77b11640 -1 reset not still connected to 0x5649d7389c70 2024-08-31T22:39:34.911 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:34.913+0000 7f1a77b11640 -1 reset not still connected to 0x5649dca20680 2024-08-31T22:39:34.912 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:34.913+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfc38000 2024-08-31T22:39:34.912 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:34.913+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:39:34.912 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:34.913+0000 7f1a77b11640 -1 reset not still connected to 0x5649e235e680 2024-08-31T22:39:34.912 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:34.913+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:39:34.912 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:34.913+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:39:34.912 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:34.913+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:39:35.329 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:35.334+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff16d00 2024-08-31T22:39:35.330 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:35.334+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff17c70 2024-08-31T22:39:35.435 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:35.433+0000 7f1a77b11640 -1 reset not still connected to 0x5649dd3d6a90 2024-08-31T22:39:35.436 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:35.433+0000 7f1a77b11640 -1 reset not still connected to 0x5649e11c6000 2024-08-31T22:39:35.436 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:35.433+0000 7f1a77b11640 -1 reset not still connected to 0x5649f276dee0 2024-08-31T22:39:35.772 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:35.769+0000 7f1a77b11640 -1 reset not still connected to 0x5649d2b4e4e0 2024-08-31T22:39:35.773 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:35.773+0000 7f1a77b11640 -1 reset not still connected to 0x5649d7389c70 2024-08-31T22:39:35.775 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:35.773+0000 7f1a77b11640 -1 reset not still connected to 0x5649dca20680 2024-08-31T22:39:35.775 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:35.773+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfc38000 2024-08-31T22:39:35.775 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:35.773+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:39:35.784 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:35.781+0000 7f1a77b11640 -1 reset not still connected to 0x5649e235e680 2024-08-31T22:39:35.784 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:35.781+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:39:35.784 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:35.781+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:39:35.784 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:35.781+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:39:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 61630464 bytes 2024-08-31T22:39:37.009 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:37.009+0000 7f1a77b11640 -1 reset not still connected to 0x5649c0c960d0 2024-08-31T22:39:37.009 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:37.009+0000 7f1a77b11640 -1 reset not still connected to 0x5649c0c96b60 2024-08-31T22:39:37.009 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:37.009+0000 7f1a77b11640 -1 reset not still connected to 0x5649d2b4e4e0 2024-08-31T22:39:37.009 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:37.009+0000 7f1a77b11640 -1 reset not still connected to 0x5649d7389c70 2024-08-31T22:39:37.015 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:37.017+0000 7f1a77b11640 -1 reset not still connected to 0x5649dca20680 2024-08-31T22:39:37.015 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:37.017+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfc38000 2024-08-31T22:39:37.016 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:37.017+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:39:37.016 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:37.017+0000 7f1a77b11640 -1 reset not still connected to 0x5649e235e680 2024-08-31T22:39:37.016 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:37.017+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:39:37.016 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:37.017+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:39:37.016 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:37.017+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:39:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 104673280 bytes 2024-08-31T22:39:40.254 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:40.253+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:39:40.254 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:40.253+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:39:40.783 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:40.785+0000 7f338ab71640 -1 reset not still connected to 0x555ce2638a90 2024-08-31T22:39:40.784 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:40.785+0000 7f338ab71640 -1 reset not still connected to 0x555ce293a410 2024-08-31T22:39:40.784 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:40.785+0000 7f338ab71640 -1 reset not still connected to 0x555ce66a0dd0 2024-08-31T22:39:40.784 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:40.785+0000 7f338ab71640 -1 reset not still connected to 0x555d168d1790 2024-08-31T22:39:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:39:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:39:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:39:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:39:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:39:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_45 2024-08-31T22:39:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 journal 128M 2024-08-31T22:39:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:39:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_45 2024-08-31T22:39:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:39:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:39:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_45 128M 2024-08-31T22:39:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:39:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_45 2024-08-31T22:39:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:39:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:39:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:39:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_45 2024-08-31T22:39:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:39:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:39:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:39:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:39:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:39:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_45 2024-08-31T22:39:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:39:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_45 2024-08-31T22:39:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:39:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=8 2024-08-31T22:39:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:39:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:39:44.102 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:44.101+0000 7f1a77b11640 -1 reset not still connected to 0x5649c0752750 2024-08-31T22:39:44.102 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:44.101+0000 7f1a77b11640 -1 reset not still connected to 0x5649c0c960d0 2024-08-31T22:39:44.102 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:44.101+0000 7f1a77b11640 -1 reset not still connected to 0x5649c0c96b60 2024-08-31T22:39:44.104 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:44.101+0000 7f1a77b11640 -1 reset not still connected to 0x5649caa220d0 2024-08-31T22:39:44.104 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:44.101+0000 7f1a77b11640 -1 reset not still connected to 0x5649d2b4e4e0 2024-08-31T22:39:44.104 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:44.101+0000 7f1a77b11640 -1 reset not still connected to 0x5649d3f1c340 2024-08-31T22:39:44.104 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:44.101+0000 7f1a77b11640 -1 reset not still connected to 0x5649d7389c70 2024-08-31T22:39:44.104 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:44.101+0000 7f1a77b11640 -1 reset not still connected to 0x5649dca20680 2024-08-31T22:39:44.109 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:44.109+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfc38000 2024-08-31T22:39:44.109 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:44.109+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:39:44.109 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:44.109+0000 7f1a77b11640 -1 reset not still connected to 0x5649e235e680 2024-08-31T22:39:44.109 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:44.109+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:39:44.109 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:44.109+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:39:44.109 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:39:44.109+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:39:45.564 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:45.566+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff161a0 2024-08-31T22:39:45.564 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:45.566+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff169c0 2024-08-31T22:39:45.573 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:45.570+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff16d00 2024-08-31T22:39:45.573 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:45.570+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff17c70 2024-08-31T22:39:45.586 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:45.585+0000 7f338ab71640 -1 reset not still connected to 0x555ce293a000 2024-08-31T22:39:45.625 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:45.625+0000 7f338ab71640 -1 reset not still connected to 0x555ce293a000 2024-08-31T22:39:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 27776000 bytes 2024-08-31T22:39:46.490 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:46.489+0000 7f338ab71640 -1 reset not still connected to 0x555ce28676c0 2024-08-31T22:39:46.497 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:46.493+0000 7f338ab71640 -1 reset not still connected to 0x555ce2867a00 2024-08-31T22:39:46.501 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:46.493+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:39:46.501 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:46.493+0000 7f338ab71640 -1 reset not still connected to 0x555cf25991e0 2024-08-31T22:39:46.505 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:46.497+0000 7f338ab71640 -1 reset not still connected to 0x555cf7f8f450 2024-08-31T22:39:46.505 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:46.497+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:39:46.505 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:46.497+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:39:46.506 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:46.505+0000 7f338ab71640 -1 reset not still connected to 0x555d0eeeaa90 2024-08-31T22:39:46.506 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:46.505+0000 7f338ab71640 -1 reset not still connected to 0x555d1114e8f0 2024-08-31T22:39:46.506 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:46.505+0000 7f338ab71640 -1 reset not still connected to 0x555d134d2000 2024-08-31T22:39:46.507 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:46.509+0000 7f338ab71640 -1 reset not still connected to 0x555d1c89bd40 2024-08-31T22:39:46.507 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:46.509+0000 7f338ab71640 -1 reset not still connected to 0x555d1cc7c750 2024-08-31T22:39:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 52862976 bytes 2024-08-31T22:39:48.300 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:48.301+0000 7f13efcb9640 -1 reset not still connected to 0x559f13dc5040 2024-08-31T22:39:48.300 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:48.301+0000 7f13efcb9640 -1 reset not still connected to 0x559f14700b60 2024-08-31T22:39:48.300 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:48.301+0000 7f13efcb9640 -1 reset not still connected to 0x559f19a8ed00 2024-08-31T22:39:48.300 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:48.301+0000 7f13efcb9640 -1 reset not still connected to 0x559f21abe410 2024-08-31T22:39:48.300 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:48.301+0000 7f13efcb9640 -1 reset not still connected to 0x559f2246c750 2024-08-31T22:39:48.300 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:48.301+0000 7f13efcb9640 -1 reset not still connected to 0x559f24292410 2024-08-31T22:39:48.301 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:48.301+0000 7f13efcb9640 -1 reset not still connected to 0x559f287c15f0 2024-08-31T22:39:48.301 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:48.301+0000 7f13efcb9640 -1 reset not still connected to 0x559f2b7404e0 2024-08-31T22:39:48.303 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:48.305+0000 7f13efcb9640 -1 reset not still connected to 0x559f2d6d9c70 2024-08-31T22:39:48.304 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:39:48.305+0000 7f13efcb9640 -1 reset not still connected to 0x559f2f205d40 2024-08-31T22:39:48.435 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:48.433+0000 7f338ab71640 -1 reset not still connected to 0x555ce2638a90 2024-08-31T22:39:48.437 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:48.437+0000 7f338ab71640 -1 reset not still connected to 0x555ce26392b0 2024-08-31T22:39:48.438 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:48.437+0000 7f338ab71640 -1 reset not still connected to 0x555ce2866000 2024-08-31T22:39:48.438 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:48.437+0000 7f338ab71640 -1 reset not still connected to 0x555ce293a410 2024-08-31T22:39:48.438 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:48.437+0000 7f338ab71640 -1 reset not still connected to 0x555ce66a0dd0 2024-08-31T22:39:48.438 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:48.437+0000 7f338ab71640 -1 reset not still connected to 0x555cea62d5f0 2024-08-31T22:39:48.438 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:48.437+0000 7f338ab71640 -1 reset not still connected to 0x555cfad9f040 2024-08-31T22:39:48.441 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:48.441+0000 7f338ab71640 -1 reset not still connected to 0x555d0a5768f0 2024-08-31T22:39:48.441 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:48.441+0000 7f338ab71640 -1 reset not still connected to 0x555d168d1790 2024-08-31T22:39:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 69609472 bytes 2024-08-31T22:39:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:39:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:39:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:39:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:39:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:39:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_46 2024-08-31T22:39:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 journal 128M 2024-08-31T22:39:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:39:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_46 2024-08-31T22:39:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:39:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:39:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_46 128M 2024-08-31T22:39:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:39:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_46 2024-08-31T22:39:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:39:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:39:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:39:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:39:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_46 2024-08-31T22:39:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:39:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:39:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:39:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:39:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:39:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_46 2024-08-31T22:39:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:39:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:39:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_46 2024-08-31T22:39:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:39:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=9 2024-08-31T22:39:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:39:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:39:52.668 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.669+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:39:52.669 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.669+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:39:52.677 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.677+0000 7f6588f42640 -1 reset not still connected to 0x562030bca340 2024-08-31T22:39:52.678 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.677+0000 7f6588f42640 -1 reset not still connected to 0x5620324664e0 2024-08-31T22:39:52.678 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.677+0000 7f6588f42640 -1 reset not still connected to 0x562037ec7ee0 2024-08-31T22:39:52.679 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.677+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:39:52.680 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.677+0000 7f6588f42640 -1 reset not still connected to 0x5620404e49c0 2024-08-31T22:39:52.680 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.677+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:39:52.680 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.681+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:39:52.680 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.681+0000 7f6588f42640 -1 reset not still connected to 0x562045a6a270 2024-08-31T22:39:52.680 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.681+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:39:52.680 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.681+0000 7f6588f42640 -1 reset not still connected to 0x562046a39380 2024-08-31T22:39:52.683 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.685+0000 7f6588f42640 -1 reset not still connected to 0x562046c82ea0 2024-08-31T22:39:52.684 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.685+0000 7f6588f42640 -1 reset not still connected to 0x562046c83860 2024-08-31T22:39:52.684 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.685+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:39:52.684 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.685+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:39:52.684 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.685+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:39:52.684 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.685+0000 7f6588f42640 -1 reset not still connected to 0x562051276340 2024-08-31T22:39:52.684 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.685+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:39:52.684 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.685+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:39:52.684 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.685+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:39:52.684 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.685+0000 7f6588f42640 -1 reset not still connected to 0x562058e2bc70 2024-08-31T22:39:52.684 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.685+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:39:52.684 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:39:52.685+0000 7f6588f42640 -1 reset not still connected to 0x56205d712b60 2024-08-31T22:39:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 57211904 bytes 2024-08-31T22:39:56.473 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:39:56.473+0000 7f338ab71640 -1 reset not still connected to 0x555d168d1790 2024-08-31T22:40:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 132093952 bytes 2024-08-31T22:40:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:40:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:40:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:40:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:40:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:40:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_47 2024-08-31T22:40:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 journal 128M 2024-08-31T22:40:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:40:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:40:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_47 2024-08-31T22:40:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:40:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:40:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_47 128M 2024-08-31T22:40:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:40:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:40:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_47 2024-08-31T22:40:00.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:00.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:40:00.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:40:00.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:40:00.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:00.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_47 2024-08-31T22:40:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:40:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:40:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:40:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:40:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:40:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:40:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_47 2024-08-31T22:40:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:40:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:40:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_47 2024-08-31T22:40:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:40:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=6 2024-08-31T22:40:01.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:40:01.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:40:01.345 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:01.345+0000 7f338ab71640 -1 reset not still connected to 0x555ce26391e0 2024-08-31T22:40:01.345 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:01.345+0000 7f338ab71640 -1 reset not still connected to 0x555ce2673450 2024-08-31T22:40:01.345 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:01.345+0000 7f338ab71640 -1 reset not still connected to 0x555ce28676c0 2024-08-31T22:40:01.345 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:01.345+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:40:01.345 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:01.345+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:40:01.633 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.629+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:40:01.633 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.629+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:40:01.642 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.641+0000 7f6588f42640 -1 reset not still connected to 0x562030bca340 2024-08-31T22:40:01.642 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.641+0000 7f6588f42640 -1 reset not still connected to 0x5620324664e0 2024-08-31T22:40:01.642 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.641+0000 7f6588f42640 -1 reset not still connected to 0x562037ec7ee0 2024-08-31T22:40:01.642 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.641+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:40:01.643 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.641+0000 7f6588f42640 -1 reset not still connected to 0x5620404e49c0 2024-08-31T22:40:01.643 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.641+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:40:01.643 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.641+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:40:01.643 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.641+0000 7f6588f42640 -1 reset not still connected to 0x562045a6a270 2024-08-31T22:40:01.643 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.641+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:40:01.648 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.649+0000 7f6588f42640 -1 reset not still connected to 0x562046a39380 2024-08-31T22:40:01.648 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.649+0000 7f6588f42640 -1 reset not still connected to 0x562046c82ea0 2024-08-31T22:40:01.648 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.649+0000 7f6588f42640 -1 reset not still connected to 0x562046c83860 2024-08-31T22:40:01.648 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.649+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:40:01.648 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.649+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:40:01.648 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.649+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:40:01.648 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.649+0000 7f6588f42640 -1 reset not still connected to 0x562051276340 2024-08-31T22:40:01.649 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.649+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:40:01.649 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.649+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:40:01.649 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.649+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:40:01.649 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.649+0000 7f6588f42640 -1 reset not still connected to 0x562058e2bc70 2024-08-31T22:40:01.649 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.649+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:40:01.649 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:01.649+0000 7f6588f42640 -1 reset not still connected to 0x56205d712b60 2024-08-31T22:40:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 24742912 bytes 2024-08-31T22:40:05.070 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:05.069+0000 7f6588f42640 -1 reset not still connected to 0x562032a40680 2024-08-31T22:40:05.070 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:05.069+0000 7f6588f42640 -1 reset not still connected to 0x562032a40d00 2024-08-31T22:40:05.070 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:05.069+0000 7f6588f42640 -1 reset not still connected to 0x562037235a00 2024-08-31T22:40:05.077 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:05.077+0000 7f6588f42640 -1 reset not still connected to 0x562037adec30 2024-08-31T22:40:05.077 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:05.077+0000 7f6588f42640 -1 reset not still connected to 0x562038de0a90 2024-08-31T22:40:05.077 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:05.077+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:40:05.077 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:05.077+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:40:05.079 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:05.077+0000 7f6588f42640 -1 reset not still connected to 0x562045dee270 2024-08-31T22:40:05.079 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:05.077+0000 7f6588f42640 -1 reset not still connected to 0x56204cad00d0 2024-08-31T22:40:05.079 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:05.077+0000 7f6588f42640 -1 reset not still connected to 0x562054033ee0 2024-08-31T22:40:05.080 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:05.077+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:40:05.080 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:05.077+0000 7f6588f42640 -1 reset not still connected to 0x562057bb5040 2024-08-31T22:40:05.080 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:05.077+0000 7f6588f42640 -1 reset not still connected to 0x562058e9be10 2024-08-31T22:40:05.080 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:05.077+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:40:05.080 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:05.077+0000 7f6588f42640 -1 reset not still connected to 0x56205b8c0ea0 2024-08-31T22:40:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 38274048 bytes 2024-08-31T22:40:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 52458496 bytes 2024-08-31T22:40:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:40:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:40:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:40:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:40:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:40:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_48 2024-08-31T22:40:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 journal 128M 2024-08-31T22:40:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:40:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:40:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_48 2024-08-31T22:40:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:40:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:40:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_48 128M 2024-08-31T22:40:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:40:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:40:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_48 2024-08-31T22:40:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:40:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:40:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:40:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_48 2024-08-31T22:40:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:40:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:40:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:40:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:40:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:40:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:40:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_48 2024-08-31T22:40:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:40:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:40:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_48 2024-08-31T22:40:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:40:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=6 2024-08-31T22:40:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:40:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:40:08.900 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:08.893+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:40:08.900 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:08.893+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:40:08.902 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:08.901+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:40:08.902 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:08.901+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:40:08.902 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:08.901+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:40:08.908 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:08.909+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:40:08.909 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:08.909+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:40:08.909 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:08.909+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:40:08.909 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:08.909+0000 7f6588f42640 -1 reset not still connected to 0x562058e2bc70 2024-08-31T22:40:08.909 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:08.909+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:40:08.909 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:08.909+0000 7f6588f42640 -1 reset not still connected to 0x56205d712b60 2024-08-31T22:40:09.728 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:09.721+0000 7f6588f42640 -1 reset not still connected to 0x562032a40d00 2024-08-31T22:40:09.728 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:09.721+0000 7f6588f42640 -1 reset not still connected to 0x562037235a00 2024-08-31T22:40:09.728 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:09.721+0000 7f6588f42640 -1 reset not still connected to 0x562037adec30 2024-08-31T22:40:09.728 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:09.721+0000 7f6588f42640 -1 reset not still connected to 0x562038de0a90 2024-08-31T22:40:09.732 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:09.729+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:40:09.732 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:09.729+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:40:09.732 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:09.729+0000 7f6588f42640 -1 reset not still connected to 0x562045dee270 2024-08-31T22:40:09.732 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:09.729+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:40:09.732 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:09.729+0000 7f6588f42640 -1 reset not still connected to 0x562057bb5040 2024-08-31T22:40:09.732 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:09.729+0000 7f6588f42640 -1 reset not still connected to 0x562058e9be10 2024-08-31T22:40:09.732 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:09.729+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:40:09.732 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:09.729+0000 7f6588f42640 -1 reset not still connected to 0x56205b8c0ea0 2024-08-31T22:40:09.732 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:09.729+0000 7f6588f42640 -1 reset not still connected to 0x56202997c9c0 2024-08-31T22:40:09.732 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:09.729+0000 7f6588f42640 -1 reset not still connected to 0x562037adec30 2024-08-31T22:40:09.732 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:09.729+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:40:09.732 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:09.729+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:40:09.733 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:09.729+0000 7f6588f42640 -1 reset not still connected to 0x562057bb5040 2024-08-31T22:40:09.733 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:09.729+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:40:10.912 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:10.909+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfc38000 2024-08-31T22:40:10.912 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:10.909+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:40:10.915 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:10.913+0000 7f1a77b11640 -1 reset not still connected to 0x5649e235e680 2024-08-31T22:40:10.915 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:10.913+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:40:10.915 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:10.913+0000 7f1a77b11640 -1 reset not still connected to 0x5649f2c1f790 2024-08-31T22:40:10.915 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:10.913+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:40:10.915 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:10.913+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:40:11.957 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:11.957+0000 7f338ab71640 -1 reset not still connected to 0x555ce2638000 2024-08-31T22:40:11.957 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:11.957+0000 7f338ab71640 -1 reset not still connected to 0x555ce26391e0 2024-08-31T22:40:11.958 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:11.957+0000 7f338ab71640 -1 reset not still connected to 0x555ce2673450 2024-08-31T22:40:11.958 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:11.957+0000 7f338ab71640 -1 reset not still connected to 0x555ce28676c0 2024-08-31T22:40:11.969 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:11.969+0000 7f338ab71640 -1 reset not still connected to 0x555ce2867a00 2024-08-31T22:40:11.969 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:11.969+0000 7f338ab71640 -1 reset not still connected to 0x555ce58b2b60 2024-08-31T22:40:11.969 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:11.969+0000 7f338ab71640 -1 reset not still connected to 0x555cee1ea8f0 2024-08-31T22:40:11.969 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:11.969+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:40:11.969 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:11.969+0000 7f338ab71640 -1 reset not still connected to 0x555cf25991e0 2024-08-31T22:40:11.969 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:11.969+0000 7f338ab71640 -1 reset not still connected to 0x555cf7f8f450 2024-08-31T22:40:11.969 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:11.969+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:40:11.969 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:11.969+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:40:11.969 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:11.969+0000 7f338ab71640 -1 reset not still connected to 0x555d0eeeaa90 2024-08-31T22:40:11.969 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:11.969+0000 7f338ab71640 -1 reset not still connected to 0x555d1114e8f0 2024-08-31T22:40:11.969 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:11.969+0000 7f338ab71640 -1 reset not still connected to 0x555d13b1d110 2024-08-31T22:40:11.969 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:11.969+0000 7f338ab71640 -1 reset not still connected to 0x555d1c89bd40 2024-08-31T22:40:11.970 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:11.969+0000 7f338ab71640 -1 reset not still connected to 0x555d1cc7c750 2024-08-31T22:40:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:40:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:40:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:40:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:40:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:40:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_49 2024-08-31T22:40:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 journal 128M 2024-08-31T22:40:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:40:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:40:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_49 2024-08-31T22:40:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:40:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:40:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_49 128M 2024-08-31T22:40:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:40:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:40:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_49 2024-08-31T22:40:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:40:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:40:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:40:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_49 2024-08-31T22:40:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:40:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:40:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:40:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:40:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:40:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:40:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_49 2024-08-31T22:40:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:40:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:40:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_49 2024-08-31T22:40:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:40:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=13 2024-08-31T22:40:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:40:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:40:13.553 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:13.553+0000 7f1a77b11640 -1 reset not still connected to 0x5649d7389c70 2024-08-31T22:40:13.557 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:13.557+0000 7f1a77b11640 -1 reset not still connected to 0x5649dca20680 2024-08-31T22:40:13.561 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:13.561+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:40:13.561 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:13.561+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:40:13.561 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:13.561+0000 7f1a77b11640 -1 reset not still connected to 0x5649f2c1f790 2024-08-31T22:40:13.561 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:13.561+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:40:13.561 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:13.561+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:40:14.987 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:14.985+0000 7f338ab71640 -1 reset not still connected to 0x555ce133fe10 2024-08-31T22:40:14.994 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:14.993+0000 7f338ab71640 -1 reset not still connected to 0x555ce2638000 2024-08-31T22:40:14.994 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:14.993+0000 7f338ab71640 -1 reset not still connected to 0x555ce26391e0 2024-08-31T22:40:14.994 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:14.993+0000 7f338ab71640 -1 reset not still connected to 0x555ce2673450 2024-08-31T22:40:14.994 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:14.993+0000 7f338ab71640 -1 reset not still connected to 0x555ce28676c0 2024-08-31T22:40:14.999 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:14.997+0000 7f338ab71640 -1 reset not still connected to 0x555ce2867a00 2024-08-31T22:40:14.999 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:14.997+0000 7f338ab71640 -1 reset not still connected to 0x555ce58b2b60 2024-08-31T22:40:14.999 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:14.997+0000 7f338ab71640 -1 reset not still connected to 0x555cee1ea8f0 2024-08-31T22:40:14.999 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:14.997+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:40:14.999 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:14.997+0000 7f338ab71640 -1 reset not still connected to 0x555cf7f8f450 2024-08-31T22:40:14.999 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:14.997+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:40:14.999 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:14.997+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:40:14.999 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:14.997+0000 7f338ab71640 -1 reset not still connected to 0x555d0eeeaa90 2024-08-31T22:40:15.000 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:14.997+0000 7f338ab71640 -1 reset not still connected to 0x555d1114e8f0 2024-08-31T22:40:15.000 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:14.997+0000 7f338ab71640 -1 reset not still connected to 0x555d13b1d110 2024-08-31T22:40:15.000 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:14.997+0000 7f338ab71640 -1 reset not still connected to 0x555d1c89bd40 2024-08-31T22:40:15.000 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:14.997+0000 7f338ab71640 -1 reset not still connected to 0x555d1cc7c750 2024-08-31T22:40:16.390 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x562030bca340 2024-08-31T22:40:16.390 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x5620324664e0 2024-08-31T22:40:16.390 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x562037ec7ee0 2024-08-31T22:40:16.390 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:40:16.390 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x5620404e49c0 2024-08-31T22:40:16.390 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:40:16.390 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:40:16.391 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x562045a6a270 2024-08-31T22:40:16.391 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:40:16.392 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x562046a39380 2024-08-31T22:40:16.392 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x562046c82ea0 2024-08-31T22:40:16.392 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x562046c83860 2024-08-31T22:40:16.392 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:40:16.392 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:40:16.392 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:40:16.393 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x562051276340 2024-08-31T22:40:16.393 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:40:16.393 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:40:16.393 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:40:16.393 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x562058e2bc70 2024-08-31T22:40:16.393 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:40:16.393 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:16.389+0000 7f6588f42640 -1 reset not still connected to 0x56205d712b60 2024-08-31T22:40:16.623 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:16.621+0000 7f1a77b11640 -1 reset not still connected to 0x5649dd3d6a90 2024-08-31T22:40:16.623 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:16.621+0000 7f1a77b11640 -1 reset not still connected to 0x5649e11c6000 2024-08-31T22:40:16.623 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:16.621+0000 7f1a77b11640 -1 reset not still connected to 0x5649eb5221a0 2024-08-31T22:40:16.623 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:16.621+0000 7f1a77b11640 -1 reset not still connected to 0x5649f276dee0 2024-08-31T22:40:17.123 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:17.121+0000 7f1a77b11640 -1 reset not still connected to 0x5649beff6270 2024-08-31T22:40:17.123 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:17.121+0000 7f1a77b11640 -1 reset not still connected to 0x5649d2103ee0 2024-08-31T22:40:17.123 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:17.121+0000 7f1a77b11640 -1 reset not still connected to 0x5649dc12add0 2024-08-31T22:40:17.123 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:17.121+0000 7f1a77b11640 -1 reset not still connected to 0x5649dc64c270 2024-08-31T22:40:17.131 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:17.133+0000 7f1a77b11640 -1 reset not still connected to 0x5649dd3d6a90 2024-08-31T22:40:17.131 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:17.133+0000 7f1a77b11640 -1 reset not still connected to 0x5649e11c6000 2024-08-31T22:40:17.139 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:17.137+0000 7f1a77b11640 -1 reset not still connected to 0x5649eb5221a0 2024-08-31T22:40:17.139 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:17.137+0000 7f1a77b11640 -1 reset not still connected to 0x5649f276dee0 2024-08-31T22:40:17.139 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:17.137+0000 7f1a77b11640 -1 reset not still connected to 0x5649f41d55f0 2024-08-31T22:40:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 84503552 bytes 2024-08-31T22:40:20.925 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:40:20.917+0000 7f13efcb9640 -1 reset not still connected to 0x559f287c15f0 2024-08-31T22:40:20.925 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:40:20.925+0000 7f13efcb9640 -1 reset not still connected to 0x559f2b7404e0 2024-08-31T22:40:20.928 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:40:20.929+0000 7f13efcb9640 -1 reset not still connected to 0x559f2d6d9c70 2024-08-31T22:40:20.928 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:40:20.929+0000 7f13efcb9640 -1 reset not still connected to 0x559f2f205d40 2024-08-31T22:40:20.929 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:40:20.929+0000 7f13efcb9640 -1 reset not still connected to 0x559f363b1790 2024-08-31T22:40:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 127981568 bytes 2024-08-31T22:40:22.709 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.709+0000 7f6588f42640 -1 reset not still connected to 0x56202611d040 2024-08-31T22:40:22.709 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.709+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:40:22.709 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.709+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:40:22.709 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.709+0000 7f6588f42640 -1 reset not still connected to 0x5620324664e0 2024-08-31T22:40:22.710 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.709+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:40:22.718 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.717+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:40:22.718 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.717+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:40:22.718 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.717+0000 7f6588f42640 -1 reset not still connected to 0x562045a6a270 2024-08-31T22:40:22.718 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.717+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:40:22.718 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.717+0000 7f6588f42640 -1 reset not still connected to 0x562046c82ea0 2024-08-31T22:40:22.719 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.717+0000 7f6588f42640 -1 reset not still connected to 0x562046c83860 2024-08-31T22:40:22.719 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.717+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:40:22.719 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.717+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:40:22.719 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.717+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:40:22.719 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.717+0000 7f6588f42640 -1 reset not still connected to 0x562051047d40 2024-08-31T22:40:22.719 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.717+0000 7f6588f42640 -1 reset not still connected to 0x562051276340 2024-08-31T22:40:22.719 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.717+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:40:22.719 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.717+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:40:22.719 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.717+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:40:22.719 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.717+0000 7f6588f42640 -1 reset not still connected to 0x562058e2bc70 2024-08-31T22:40:22.719 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.717+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:40:22.719 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.717+0000 7f6588f42640 -1 reset not still connected to 0x56205d712b60 2024-08-31T22:40:22.832 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.833+0000 7f6588f42640 -1 reset not still connected to 0x562032a40d00 2024-08-31T22:40:22.832 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.833+0000 7f6588f42640 -1 reset not still connected to 0x562037235a00 2024-08-31T22:40:22.832 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.833+0000 7f6588f42640 -1 reset not still connected to 0x562037adec30 2024-08-31T22:40:22.832 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.833+0000 7f6588f42640 -1 reset not still connected to 0x562038de0a90 2024-08-31T22:40:22.835 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.833+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:40:22.835 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.833+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:40:22.835 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.833+0000 7f6588f42640 -1 reset not still connected to 0x562045dee270 2024-08-31T22:40:22.835 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.833+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:40:22.835 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.833+0000 7f6588f42640 -1 reset not still connected to 0x562057bb5040 2024-08-31T22:40:22.835 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.833+0000 7f6588f42640 -1 reset not still connected to 0x562058e9be10 2024-08-31T22:40:22.835 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.833+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:40:22.837 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:22.833+0000 7f6588f42640 -1 reset not still connected to 0x56205b8c0ea0 2024-08-31T22:40:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 149010432 bytes 2024-08-31T22:40:24.839 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:24.833+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:40:24.842 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:24.833+0000 7f1a77b11640 -1 reset not still connected to 0x5649e235e680 2024-08-31T22:40:24.842 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:24.833+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:40:24.842 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:24.837+0000 7f1a77b11640 -1 reset not still connected to 0x5649f2c1f790 2024-08-31T22:40:24.843 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:24.837+0000 7f1a77b11640 -1 reset not still connected to 0x5649f6ebcd00 2024-08-31T22:40:24.843 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:24.837+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:40:24.843 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:24.837+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:40:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 181749760 bytes 2024-08-31T22:40:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:40:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:40:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:40:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:40:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:40:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_50 2024-08-31T22:40:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 journal 128M 2024-08-31T22:40:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:40:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:40:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_50 2024-08-31T22:40:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:40:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n journal ']' 2024-08-31T22:40:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_50 128M 2024-08-31T22:40:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:40:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:40:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_50 2024-08-31T22:40:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:40:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 128M ']' 2024-08-31T22:40:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=128M 2024-08-31T22:40:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:40:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_50 2024-08-31T22:40:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ is_pool_mirror_mode_image mirror 2024-08-31T22:40:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:40:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' pool = image ']' 2024-08-31T22:40:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ case "${pool}" in 2024-08-31T22:40:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 1 2024-08-31T22:40:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n '' ']' 2024-08-31T22:40:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_50 2024-08-31T22:40:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:40:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:40:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_50 2024-08-31T22:40:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:40:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=6 2024-08-31T22:40:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:40:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:40:26.742 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:26.741+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:40:26.742 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:26.741+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:40:26.742 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:26.741+0000 7f1a77b11640 -1 reset not still connected to 0x5649f2c1f790 2024-08-31T22:40:26.757 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:26.753+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:40:26.758 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:40:26.753+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:40:27.430 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:40:27.433+0000 7fdf7b115640 -1 reset not still connected to 0x55dde4411450 2024-08-31T22:40:27.430 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:40:27.433+0000 7fdf7b115640 -1 reset not still connected to 0x55dde49f60d0 2024-08-31T22:40:27.430 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:40:27.433+0000 7fdf7b115640 -1 reset not still connected to 0x55ddf45240d0 2024-08-31T22:40:27.430 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:40:27.433+0000 7fdf7b115640 -1 reset not still connected to 0x55ddfb2c9ad0 2024-08-31T22:40:30.406 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:40:30.409+0000 7fdf7b115640 -1 reset not still connected to 0x55ddf45240d0 2024-08-31T22:40:30.939 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:30.928+0000 7f338ab71640 -1 reset not still connected to 0x555cf7f8f450 2024-08-31T22:40:30.940 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:30.928+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:40:30.940 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:30.928+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:40:30.940 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:30.928+0000 7f338ab71640 -1 reset not still connected to 0x555d0eeeaa90 2024-08-31T22:40:30.940 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:30.928+0000 7f338ab71640 -1 reset not still connected to 0x555d1114e8f0 2024-08-31T22:40:30.945 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:30.944+0000 7f338ab71640 -1 reset not still connected to 0x555d13b1d110 2024-08-31T22:40:30.956 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:30.956+0000 7f338ab71640 -1 reset not still connected to 0x555d1c89bd40 2024-08-31T22:40:30.956 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:30.956+0000 7f338ab71640 -1 reset not still connected to 0x555d1cc7c750 2024-08-31T22:40:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 61248512 bytes 2024-08-31T22:40:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 62224384 bytes 2024-08-31T22:40:31.434 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:31.428+0000 7f338ab71640 -1 reset not still connected to 0x555ce28676c0 2024-08-31T22:40:31.434 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:31.432+0000 7f338ab71640 -1 reset not still connected to 0x555ce2867a00 2024-08-31T22:40:31.434 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:31.432+0000 7f338ab71640 -1 reset not still connected to 0x555ce58b2b60 2024-08-31T22:40:31.434 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:31.432+0000 7f338ab71640 -1 reset not still connected to 0x555ce619a680 2024-08-31T22:40:31.446 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:31.444+0000 7f338ab71640 -1 reset not still connected to 0x555ceb8d12b0 2024-08-31T22:40:31.446 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:31.444+0000 7f338ab71640 -1 reset not still connected to 0x555cee1ea8f0 2024-08-31T22:40:31.446 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:31.444+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:40:31.446 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:31.444+0000 7f338ab71640 -1 reset not still connected to 0x555cf25991e0 2024-08-31T22:40:31.446 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:31.444+0000 7f338ab71640 -1 reset not still connected to 0x555cf7f8f450 2024-08-31T22:40:31.446 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:31.444+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:40:31.446 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:31.444+0000 7f338ab71640 -1 reset not still connected to 0x555d0b8de000 2024-08-31T22:40:31.446 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:31.444+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:40:31.446 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:31.444+0000 7f338ab71640 -1 reset not still connected to 0x555d0eeeaa90 2024-08-31T22:40:31.447 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:31.444+0000 7f338ab71640 -1 reset not still connected to 0x555d1114e8f0 2024-08-31T22:40:31.447 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:31.444+0000 7f338ab71640 -1 reset not still connected to 0x555d134d2000 2024-08-31T22:40:31.447 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:31.444+0000 7f338ab71640 -1 reset not still connected to 0x555d13b1d110 2024-08-31T22:40:31.447 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:31.444+0000 7f338ab71640 -1 reset not still connected to 0x555d1c89bd40 2024-08-31T22:40:31.447 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:31.444+0000 7f338ab71640 -1 reset not still connected to 0x555d1cc7c750 2024-08-31T22:40:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 77328384 bytes 2024-08-31T22:40:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:40:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:40:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:40:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:40:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-08-31T22:40:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:40:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:40:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_count=50 2024-08-31T22:40:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:40:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local count 2024-08-31T22:40:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_count=0 2024-08-31T22:40:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:40:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 1 40 2024-08-31T22:40:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in `seq 1 40` 2024-08-31T22:40:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 1 -ne 1 2024-08-31T22:40:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-08-31T22:40:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep 'images: ' 2024-08-31T22:40:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut '-d ' -f 2 2024-08-31T22:40:32.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: failed to query services: (13) Permission denied 2024-08-31T22:40:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ count=50 2024-08-31T22:40:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 50 = 50 2024-08-31T22:40:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:40:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-08-31T22:40:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:40:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:40:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:40:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state 2024-08-31T22:40:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 1 40 2024-08-31T22:40:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in `seq 1 40` 2024-08-31T22:40:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 1 -ne 1 2024-08-31T22:40:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-08-31T22:40:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep 'image health:' 2024-08-31T22:40:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut '-d ' -f 3 2024-08-31T22:40:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: failed to query services: (13) Permission denied 2024-08-31T22:40:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ state=WARNING 2024-08-31T22:40:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test WARNING = ERROR 2024-08-31T22:40:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test WARNING = OK 2024-08-31T22:40:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in `seq 1 40` 2024-08-31T22:40:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 2 -ne 1 2024-08-31T22:40:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 30 2024-08-31T22:40:34.440 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:34.436+0000 7f6588f42640 -1 reset not still connected to 0x56202a17d930 2024-08-31T22:40:34.440 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:34.436+0000 7f6588f42640 -1 reset not still connected to 0x5620326af930 2024-08-31T22:40:34.443 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:34.440+0000 7f6588f42640 -1 reset not still connected to 0x562032a40680 2024-08-31T22:40:34.443 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:34.440+0000 7f6588f42640 -1 reset not still connected to 0x562032a40d00 2024-08-31T22:40:34.443 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:34.440+0000 7f6588f42640 -1 reset not still connected to 0x562036482000 2024-08-31T22:40:34.443 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:34.440+0000 7f6588f42640 -1 reset not still connected to 0x562037235a00 2024-08-31T22:40:34.445 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:34.440+0000 7f6588f42640 -1 reset not still connected to 0x562037adec30 2024-08-31T22:40:34.445 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:34.440+0000 7f6588f42640 -1 reset not still connected to 0x562038de0a90 2024-08-31T22:40:34.446 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:34.440+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:40:34.446 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:34.440+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:40:34.446 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:34.440+0000 7f6588f42640 -1 reset not still connected to 0x562045dee270 2024-08-31T22:40:34.446 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:34.440+0000 7f6588f42640 -1 reset not still connected to 0x56204cad00d0 2024-08-31T22:40:34.446 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:34.440+0000 7f6588f42640 -1 reset not still connected to 0x562054033ee0 2024-08-31T22:40:34.446 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:34.440+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:40:34.447 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:34.444+0000 7f6588f42640 -1 reset not still connected to 0x562057bb5040 2024-08-31T22:40:34.447 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:34.444+0000 7f6588f42640 -1 reset not still connected to 0x562058e9be10 2024-08-31T22:40:34.447 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:34.444+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:40:34.447 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:34.444+0000 7f6588f42640 -1 reset not still connected to 0x56205b8c0ea0 2024-08-31T22:40:42.907 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.904+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:40:42.908 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.904+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:40:42.915 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.912+0000 7f6588f42640 -1 reset not still connected to 0x5620324664e0 2024-08-31T22:40:42.917 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.916+0000 7f6588f42640 -1 reset not still connected to 0x562037ec7ee0 2024-08-31T22:40:42.917 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.916+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:40:42.917 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.916+0000 7f6588f42640 -1 reset not still connected to 0x5620404e49c0 2024-08-31T22:40:42.917 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.916+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:40:42.918 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.916+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:40:42.920 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.920+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:40:42.921 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.920+0000 7f6588f42640 -1 reset not still connected to 0x562046a39380 2024-08-31T22:40:42.921 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.920+0000 7f6588f42640 -1 reset not still connected to 0x562046c82ea0 2024-08-31T22:40:42.921 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.920+0000 7f6588f42640 -1 reset not still connected to 0x562046c83860 2024-08-31T22:40:42.921 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.920+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:40:42.921 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.920+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:40:42.922 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.920+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:40:42.923 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.924+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:40:42.923 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.924+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:40:42.923 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.924+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:40:42.923 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.924+0000 7f6588f42640 -1 reset not still connected to 0x562058e2bc70 2024-08-31T22:40:42.923 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.924+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:40:42.923 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:42.924+0000 7f6588f42640 -1 reset not still connected to 0x56205d712b60 2024-08-31T22:40:43.932 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:43.924+0000 7f338ab71640 -1 reset not still connected to 0x555cee1ea8f0 2024-08-31T22:40:43.932 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:43.924+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:40:43.932 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:43.924+0000 7f338ab71640 -1 reset not still connected to 0x555cf7f8f450 2024-08-31T22:40:43.932 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:43.924+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:40:43.932 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:43.924+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:40:43.932 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:43.924+0000 7f338ab71640 -1 reset not still connected to 0x555d0eeeaa90 2024-08-31T22:40:43.932 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:43.924+0000 7f338ab71640 -1 reset not still connected to 0x555d1114e8f0 2024-08-31T22:40:43.933 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:43.928+0000 7f338ab71640 -1 reset not still connected to 0x555d13b1d110 2024-08-31T22:40:43.933 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:43.928+0000 7f338ab71640 -1 reset not still connected to 0x555d1c89bd40 2024-08-31T22:40:43.933 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:43.928+0000 7f338ab71640 -1 reset not still connected to 0x555d1cc7c750 2024-08-31T22:40:45.706 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:45.704+0000 7f6588f42640 -1 reset not still connected to 0x56202997c9c0 2024-08-31T22:40:45.707 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:45.704+0000 7f6588f42640 -1 reset not still connected to 0x56202a17d930 2024-08-31T22:40:45.707 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:45.704+0000 7f6588f42640 -1 reset not still connected to 0x5620326af930 2024-08-31T22:40:45.712 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:45.708+0000 7f6588f42640 -1 reset not still connected to 0x562032a40680 2024-08-31T22:40:45.713 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:45.712+0000 7f6588f42640 -1 reset not still connected to 0x562032a40d00 2024-08-31T22:40:45.713 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:45.712+0000 7f6588f42640 -1 reset not still connected to 0x562037235a00 2024-08-31T22:40:45.713 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:45.712+0000 7f6588f42640 -1 reset not still connected to 0x562037adec30 2024-08-31T22:40:45.713 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:45.712+0000 7f6588f42640 -1 reset not still connected to 0x562038de0a90 2024-08-31T22:40:45.721 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:45.720+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:40:45.721 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:45.720+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:40:45.721 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:45.720+0000 7f6588f42640 -1 reset not still connected to 0x562045dee270 2024-08-31T22:40:45.721 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:45.720+0000 7f6588f42640 -1 reset not still connected to 0x56204cad00d0 2024-08-31T22:40:45.721 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:45.720+0000 7f6588f42640 -1 reset not still connected to 0x562054033ee0 2024-08-31T22:40:45.721 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:45.720+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:40:45.721 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:45.720+0000 7f6588f42640 -1 reset not still connected to 0x562057bb5040 2024-08-31T22:40:45.722 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:45.720+0000 7f6588f42640 -1 reset not still connected to 0x562058e9be10 2024-08-31T22:40:45.722 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:45.720+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:40:45.722 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:40:45.720+0000 7f6588f42640 -1 reset not still connected to 0x56205b8c0ea0 2024-08-31T22:40:48.061 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:48.060+0000 7f338ab71640 -1 reset not still connected to 0x555ce28676c0 2024-08-31T22:40:48.066 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:48.064+0000 7f338ab71640 -1 reset not still connected to 0x555ce58b2b60 2024-08-31T22:40:48.067 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:48.064+0000 7f338ab71640 -1 reset not still connected to 0x555cee1ea8f0 2024-08-31T22:40:48.067 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:48.064+0000 7f338ab71640 -1 reset not still connected to 0x555cf168fc70 2024-08-31T22:40:48.074 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:48.072+0000 7f338ab71640 -1 reset not still connected to 0x555cf7f8f450 2024-08-31T22:40:48.074 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:48.072+0000 7f338ab71640 -1 reset not still connected to 0x555cfd487790 2024-08-31T22:40:48.074 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:48.072+0000 7f338ab71640 -1 reset not still connected to 0x555d0d9deb60 2024-08-31T22:40:48.074 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:48.072+0000 7f338ab71640 -1 reset not still connected to 0x555d0eeeaa90 2024-08-31T22:40:48.074 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:48.072+0000 7f338ab71640 -1 reset not still connected to 0x555d1114e8f0 2024-08-31T22:40:48.074 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:48.072+0000 7f338ab71640 -1 reset not still connected to 0x555d13b1d110 2024-08-31T22:40:48.074 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:48.072+0000 7f338ab71640 -1 reset not still connected to 0x555d1c89bd40 2024-08-31T22:40:48.074 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:40:48.072+0000 7f338ab71640 -1 reset not still connected to 0x555d1cc7c750 2024-08-31T22:41:02.000 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:41:02.000+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfc38000 2024-08-31T22:41:02.001 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:41:02.000+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:41:02.001 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:41:02.000+0000 7f1a77b11640 -1 reset not still connected to 0x5649e72e72b0 2024-08-31T22:41:02.001 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:41:02.000+0000 7f1a77b11640 -1 reset not still connected to 0x5649f2c1f790 2024-08-31T22:41:02.007 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:41:02.004+0000 7f1a77b11640 -1 reset not still connected to 0x5649f6ebcd00 2024-08-31T22:41:02.007 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:41:02.004+0000 7f1a77b11640 -1 reset not still connected to 0x5649f76b9e10 2024-08-31T22:41:02.007 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:41:02.004+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:41:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-08-31T22:41:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep 'image health:' 2024-08-31T22:41:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut '-d ' -f 3 2024-08-31T22:41:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: failed to query services: (13) Permission denied 2024-08-31T22:41:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ state=WARNING 2024-08-31T22:41:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test WARNING = ERROR 2024-08-31T22:41:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test WARNING = OK 2024-08-31T22:41:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in `seq 1 40` 2024-08-31T22:41:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 -ne 1 2024-08-31T22:41:02.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 30 2024-08-31T22:41:09.141 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.140+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:41:09.144 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.144+0000 7f6588f42640 -1 reset not still connected to 0x56205b8c0ea0 2024-08-31T22:41:09.144 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.144+0000 7f6588f42640 -1 reset not still connected to 0x562028e1b520 2024-08-31T22:41:09.145 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.144+0000 7f6588f42640 -1 reset not still connected to 0x56202997c9c0 2024-08-31T22:41:09.145 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.144+0000 7f6588f42640 -1 reset not still connected to 0x5620326af930 2024-08-31T22:41:09.145 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.144+0000 7f6588f42640 -1 reset not still connected to 0x562032a40680 2024-08-31T22:41:09.145 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.144+0000 7f6588f42640 -1 reset not still connected to 0x562037adec30 2024-08-31T22:41:09.145 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.144+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:41:09.145 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.144+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:41:09.272 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.272+0000 7f6588f42640 -1 reset not still connected to 0x562037235a00 2024-08-31T22:41:09.272 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.272+0000 7f6588f42640 -1 reset not still connected to 0x562037adec30 2024-08-31T22:41:09.272 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.272+0000 7f6588f42640 -1 reset not still connected to 0x562038de0a90 2024-08-31T22:41:09.272 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.272+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:41:09.272 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.272+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:41:09.272 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.272+0000 7f6588f42640 -1 reset not still connected to 0x562045dee270 2024-08-31T22:41:09.272 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.272+0000 7f6588f42640 -1 reset not still connected to 0x56204cad00d0 2024-08-31T22:41:09.273 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.272+0000 7f6588f42640 -1 reset not still connected to 0x562054033ee0 2024-08-31T22:41:09.273 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.272+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:41:09.273 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.272+0000 7f6588f42640 -1 reset not still connected to 0x562057bb5040 2024-08-31T22:41:09.273 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.272+0000 7f6588f42640 -1 reset not still connected to 0x562058e9be10 2024-08-31T22:41:09.273 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:09.272+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:41:16.464 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.464+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:41:16.465 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.464+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:41:16.465 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.464+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:41:16.467 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.464+0000 7f6588f42640 -1 reset not still connected to 0x562051047d40 2024-08-31T22:41:16.470 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.468+0000 7f6588f42640 -1 reset not still connected to 0x562051276340 2024-08-31T22:41:16.470 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.468+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:41:16.470 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.468+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:41:16.470 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.468+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:41:16.471 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.468+0000 7f6588f42640 -1 reset not still connected to 0x562058e2bc70 2024-08-31T22:41:16.471 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.468+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:41:16.471 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.468+0000 7f6588f42640 -1 reset not still connected to 0x56205d712b60 2024-08-31T22:41:16.499 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.496+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:41:16.502 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.500+0000 7f6588f42640 -1 reset not still connected to 0x5620283e45b0 2024-08-31T22:41:16.502 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.500+0000 7f6588f42640 -1 reset not still connected to 0x5620289c05b0 2024-08-31T22:41:16.503 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.500+0000 7f6588f42640 -1 reset not still connected to 0x562030bca340 2024-08-31T22:41:16.503 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.500+0000 7f6588f42640 -1 reset not still connected to 0x562037ec7ee0 2024-08-31T22:41:16.504 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.504+0000 7f6588f42640 -1 reset not still connected to 0x5620404e49c0 2024-08-31T22:41:16.504 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.504+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:41:16.505 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.504+0000 7f6588f42640 -1 reset not still connected to 0x562044d884e0 2024-08-31T22:41:16.505 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.504+0000 7f6588f42640 -1 reset not still connected to 0x562045a6a270 2024-08-31T22:41:16.506 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.504+0000 7f6588f42640 -1 reset not still connected to 0x562045d5fad0 2024-08-31T22:41:16.506 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.504+0000 7f6588f42640 -1 reset not still connected to 0x562046a39380 2024-08-31T22:41:16.506 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.504+0000 7f6588f42640 -1 reset not still connected to 0x562046c83860 2024-08-31T22:41:16.506 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.504+0000 7f6588f42640 -1 reset not still connected to 0x562046dc2b60 2024-08-31T22:41:16.506 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.504+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:41:16.506 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.504+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:41:16.507 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.504+0000 7f6588f42640 -1 reset not still connected to 0x562051047d40 2024-08-31T22:41:16.507 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.504+0000 7f6588f42640 -1 reset not still connected to 0x562051276340 2024-08-31T22:41:16.507 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.504+0000 7f6588f42640 -1 reset not still connected to 0x56205340c410 2024-08-31T22:41:16.507 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.504+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:41:16.507 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.504+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:41:16.507 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.504+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:41:16.507 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:41:16.504+0000 7f6588f42640 -1 reset not still connected to 0x56205d712b60 2024-08-31T22:41:20.384 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:41:20.384+0000 7f338ab71640 -1 reset not still connected to 0x555ce2638dd0 2024-08-31T22:41:20.389 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:41:20.388+0000 7f338ab71640 -1 reset not still connected to 0x555ce28675f0 2024-08-31T22:41:20.390 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:41:20.388+0000 7f338ab71640 -1 reset not still connected to 0x555ce293a680 2024-08-31T22:41:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-08-31T22:41:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep 'image health:' 2024-08-31T22:41:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut '-d ' -f 3 2024-08-31T22:41:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: failed to query services: (13) Permission denied 2024-08-31T22:41:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ state=OK 2024-08-31T22:41:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test OK = ERROR 2024-08-31T22:41:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test OK = OK 2024-08-31T22:41:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-08-31T22:41:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_count=50 2024-08-31T22:41:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:41:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local count 2024-08-31T22:41:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_count=0 2024-08-31T22:41:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:41:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 1 40 2024-08-31T22:41:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in `seq 1 40` 2024-08-31T22:41:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 1 -ne 1 2024-08-31T22:41:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-08-31T22:41:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep 'images: ' 2024-08-31T22:41:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut '-d ' -f 2 2024-08-31T22:41:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: failed to query services: (13) Permission denied 2024-08-31T22:41:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ count=50 2024-08-31T22:41:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 50 = 50 2024-08-31T22:41:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-08-31T22:41:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:41:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state 2024-08-31T22:41:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 1 40 2024-08-31T22:41:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in `seq 1 40` 2024-08-31T22:41:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 1 -ne 1 2024-08-31T22:41:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-08-31T22:41:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep 'image health:' 2024-08-31T22:41:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut '-d ' -f 3 2024-08-31T22:41:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: failed to query services: (13) Permission denied 2024-08-31T22:41:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ state=OK 2024-08-31T22:41:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test OK = ERROR 2024-08-31T22:41:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test OK = OK 2024-08-31T22:41:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ testlog 'TEST: compare many images' 2024-08-31T22:41:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ tee -a /tmp/tmp.iYfzsjLopX/rbd-mirror.test.log 2024-08-31T22:41:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ date '+%F %T' 2024-08-31T22:41:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo 2024-08-31 22:41:33 TEST: compare many images 2024-08-31T22:41:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:2024-08-31 22:41:33 TEST: compare many images 2024-08-31T22:41:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 1 50 2024-08-31T22:41:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:41:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_1 2024-08-31T22:41:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_1 snap 2024-08-31T22:41:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:41:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:41:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:41:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:41:35.341 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:41:35.335+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff16d00 2024-08-31T22:41:35.341 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:41:35.335+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff17c70 2024-08-31T22:41:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-08-31T22:41:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:41:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-08-31T22:41:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:41:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:41:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:41:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:41:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:41:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-08-31T22:41:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:41:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:41:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:41:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:41:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-08-31T22:41:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:41:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:41:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:41:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:41:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:41:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:41:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:41:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:41:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:41:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:41:37.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:41:37.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-08-31T22:41:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:41:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:41:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:41:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:41:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:41:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:41:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:41:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:41:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:41:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:41:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:41:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:41:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:41:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:41:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:41:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:41:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:41:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:41:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:41:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:41:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:41:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:41:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:41:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:41:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:41:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:41:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:41:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-08-31T22:41:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:no valid command found; 5 closest matches: 2024-08-31T22:41:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status mirror/image_11 2024-08-31T22:41:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status mirror/image_13 2024-08-31T22:41:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status mirror/image_15 2024-08-31T22:41:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status mirror/image_17 2024-08-31T22:41:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status mirror/image_19 2024-08-31T22:41:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-08-31T22:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:41:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:41:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:41:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:41:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:41:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:41:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:41:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:41:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:41:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:41:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:41:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:41:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:41:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:41:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:41:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:41:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:41:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:41:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:41:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:41:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:41:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:41:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:41:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:41:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-08-31T22:41:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:41:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:41:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:41:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:41:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:41:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:41:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:41:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-08-31T22:41:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:41:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:41:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:41:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:41:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_1 2024-08-31T22:41:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:41:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:41:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:41:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:41:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:41:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:41:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:41:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:41:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_1 2024-08-31T22:41:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:41:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:41:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_1 ']' 2024-08-31T22:41:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_1' 2024-08-31T22:41:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_1 2024-08-31T22:41:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:41:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:41:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:41:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:41:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:41:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:41:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:41:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:41:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:41:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_1 2024-08-31T22:41:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:41:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:41:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:41:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:41:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:41:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:41:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:41:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:41:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:41:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:41:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:41:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:41:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:41:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:41:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:41:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:41:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:41:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:41:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:41:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:41:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:41:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:41:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:41:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:41:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:41:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:41:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_1 2024-08-31T22:41:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:no valid command found; 5 closest matches: 2024-08-31T22:41:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush mirror/image_11 2024-08-31T22:41:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush mirror/image_13 2024-08-31T22:41:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush mirror/image_15 2024-08-31T22:41:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush mirror/image_17 2024-08-31T22:41:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush mirror/image_19 2024-08-31T22:41:38.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:41:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:41:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_1 2024-08-31T22:41:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:41:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:41:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:41:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:41:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:41:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:41:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:41:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:41:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:41:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:41:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:41:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:41:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:41:38.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:41:38.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:41:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:41:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:41:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:41:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:41:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:41:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:41:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:41:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:41:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:41:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:41:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:41:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:41:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:41:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:41:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:41:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:41:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_1 2024-08-31T22:41:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_1 2024-08-31T22:41:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:41:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:41:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_1 2024-08-31T22:41:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_1 '' 2024-08-31T22:41:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:41:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:41:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_1 2024-08-31T22:41:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:41:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-08-31T22:41:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_1.status 2024-08-31T22:41:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:41:38.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_1.status 2024-08-31T22:41:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_1.status 2024-08-31T22:41:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-08-31T22:41:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:41:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 4 2024-08-31T22:41:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:41:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13663], [object_number=14, tag_tid=1, entry_tid=13662]]], state=connected] 2024-08-31T22:41:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13663], [object_number=14, tag_tid=1, entry_tid=13662]]], state=connected] 2024-08-31T22:41:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_1.status 2024-08-31T22:41:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2024-08-31T22:41:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_1 2024-08-31T22:41:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:41:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:41:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_1 2024-08-31T22:41:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_1 '..*' 2024-08-31T22:41:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:41:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:41:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_1 2024-08-31T22:41:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:41:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-08-31T22:41:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_1.status 2024-08-31T22:41:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:41:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_1.status 2024-08-31T22:41:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-08-31T22:41:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_1.status 2024-08-31T22:41:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:41:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 4 2024-08-31T22:41:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:41:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13663], [object_number=14, tag_tid=1, entry_tid=13662]]], state=connected] 2024-08-31T22:41:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13663], [object_number=14, tag_tid=1, entry_tid=13662]]], state=connected] 2024-08-31T22:41:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_1.status 2024-08-31T22:41:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2024-08-31T22:41:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=1' 2024-08-31T22:41:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-08-31T22:41:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:41:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:41:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:41:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:41:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:41:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-08-31T22:41:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_1': 2024-08-31T22:41:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:41:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:41:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:41:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10ab854e810f 2024-08-31T22:41:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10ab854e810f 2024-08-31T22:41:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:41:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:41:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:41:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:41:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:32:21 2024 2024-08-31T22:41:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:32:21 2024 2024-08-31T22:41:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:33:22 2024 2024-08-31T22:41:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:41:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10ab854e810f 2024-08-31T22:41:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:41:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:41:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 90868ffe-00b8-4ead-865c-fd0939ce0660 2024-08-31T22:41:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:41:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_1 snap 2024-08-31T22:41:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:41:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:41:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:41:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_1.export 2024-08-31T22:41:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_1.export 2024-08-31T22:41:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_1.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_1.export 2024-08-31T22:41:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_1.export 2024-08-31T22:41:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:41:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_1.export 2024-08-31T22:41:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:41:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_1.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_1.export 2024-08-31T22:41:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_1.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_1.export 2024-08-31T22:41:40.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:40.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:41:40.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_2 2024-08-31T22:41:40.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_2 snap 2024-08-31T22:41:40.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:41:40.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:41:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:41:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:41:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-08-31T22:41:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:41:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-08-31T22:41:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:41:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:41:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:41:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:41:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:41:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-08-31T22:41:42.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:42.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:42.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:41:42.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:41:42.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:41:42.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:41:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-08-31T22:41:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:41:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:41:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:41:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:41:42.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:41:42.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:41:42.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:41:42.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:41:42.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:41:42.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:41:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:41:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-08-31T22:41:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:41:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:41:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:41:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:41:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:41:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:41:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:41:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:41:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:41:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:41:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:41:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:41:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:41:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:41:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:41:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:41:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:41:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:41:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:41:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:41:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:41:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:41:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:41:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:41:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:41:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:41:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:41:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-08-31T22:41:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:41:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:41:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:41:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:41:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:41:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:41:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:41:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-08-31T22:41:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:41:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:41:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:41:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:41:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_2 2024-08-31T22:41:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:41:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:41:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:41:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:41:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:41:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:41:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:41:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:41:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_2 2024-08-31T22:41:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:41:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:41:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_2 ']' 2024-08-31T22:41:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_2' 2024-08-31T22:41:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_2 2024-08-31T22:41:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:41:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:41:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:41:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:41:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:41:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:41:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:41:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:41:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:41:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_2 2024-08-31T22:41:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:41:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:41:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:41:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:41:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:41:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:41:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:41:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:41:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:41:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:41:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:41:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:41:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:41:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:41:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:41:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:41:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:41:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:41:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:41:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:41:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:41:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:41:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:41:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:41:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:41:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:41:42.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_2 2024-08-31T22:41:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_2 2024-08-31T22:41:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:41:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:41:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_2 2024-08-31T22:41:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_2 '' 2024-08-31T22:41:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:41:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:41:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_2 2024-08-31T22:41:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:41:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-08-31T22:41:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_2.status 2024-08-31T22:41:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:41:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_2.status 2024-08-31T22:41:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-08-31T22:41:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_2.status 2024-08-31T22:41:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:41:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:41:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:41:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6819], [object_number=6, tag_tid=1, entry_tid=6818]]], state=connected] 2024-08-31T22:41:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6819], [object_number=6, tag_tid=1, entry_tid=6818]]], state=connected] 2024-08-31T22:41:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_2.status 2024-08-31T22:41:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-08-31T22:41:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_2 2024-08-31T22:41:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:41:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:41:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_2 2024-08-31T22:41:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_2 '..*' 2024-08-31T22:41:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:41:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:41:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_2 2024-08-31T22:41:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:41:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-08-31T22:41:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_2.status 2024-08-31T22:41:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:41:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_2.status 2024-08-31T22:41:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-08-31T22:41:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_2.status 2024-08-31T22:41:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:41:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:41:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:41:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6819], [object_number=6, tag_tid=1, entry_tid=6818]]], state=connected] 2024-08-31T22:41:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6819], [object_number=6, tag_tid=1, entry_tid=6818]]], state=connected] 2024-08-31T22:41:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_2.status 2024-08-31T22:41:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-08-31T22:41:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-08-31T22:41:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-08-31T22:41:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:41:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:41:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:41:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:41:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:41:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-08-31T22:41:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_2': 2024-08-31T22:41:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:41:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:41:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:41:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a85c21c009 2024-08-31T22:41:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a85c21c009 2024-08-31T22:41:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:41:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:41:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:41:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:41:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:32:34 2024 2024-08-31T22:41:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:32:34 2024 2024-08-31T22:41:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:33:40 2024 2024-08-31T22:41:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:41:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a85c21c009 2024-08-31T22:41:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:41:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:41:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 77ee40c8-74b7-40b4-9079-adcdfe9060d0 2024-08-31T22:41:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:41:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_2 snap 2024-08-31T22:41:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:41:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:41:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:41:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_2.export 2024-08-31T22:41:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_2.export 2024-08-31T22:41:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_2.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_2.export 2024-08-31T22:41:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_2.export 2024-08-31T22:41:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:41:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_2.export 2024-08-31T22:41:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:41:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_2.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_2.export 2024-08-31T22:41:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_2.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_2.export 2024-08-31T22:41:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:41:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_3 2024-08-31T22:41:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_3 snap 2024-08-31T22:41:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:41:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:41:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:41:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:41:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-08-31T22:41:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:41:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-08-31T22:41:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:41:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:41:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:41:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:41:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:41:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-08-31T22:41:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:41:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:41:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:41:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:41:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-08-31T22:41:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:41:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:41:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:41:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:41:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:41:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:41:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:41:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:41:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:41:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:41:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:41:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-08-31T22:41:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:41:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:41:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:41:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:41:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:41:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:41:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:41:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:41:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:41:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:41:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:41:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:41:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:41:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:41:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:41:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:41:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:41:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:41:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:41:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:41:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:41:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:41:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:41:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:41:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:41:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:41:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:41:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-08-31T22:41:48.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:41:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:41:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:41:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:41:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:41:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:41:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:41:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-08-31T22:41:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:41:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:41:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:41:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:41:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_3 2024-08-31T22:41:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:41:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:41:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:41:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:41:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:41:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:41:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:41:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:41:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_3 2024-08-31T22:41:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:41:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:41:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_3 ']' 2024-08-31T22:41:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_3' 2024-08-31T22:41:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_3 2024-08-31T22:41:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:41:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:41:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:41:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:41:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:41:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:41:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:41:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:41:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:41:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_3 2024-08-31T22:41:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:41:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:41:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:41:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:41:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:41:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:41:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:41:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:41:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:41:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:41:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:41:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:41:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:41:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:41:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:41:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:41:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:41:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:41:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:41:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:41:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:41:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:41:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:41:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:41:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:41:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:41:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_3 2024-08-31T22:41:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_3 2024-08-31T22:41:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:41:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:41:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_3 2024-08-31T22:41:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_3 '' 2024-08-31T22:41:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:41:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:41:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_3 2024-08-31T22:41:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:41:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-08-31T22:41:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_3.status 2024-08-31T22:41:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:41:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_3.status 2024-08-31T22:41:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-08-31T22:41:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_3.status 2024-08-31T22:41:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:41:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:41:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:41:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12819], [object_number=14, tag_tid=1, entry_tid=12818]]], state=connected] 2024-08-31T22:41:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12819], [object_number=14, tag_tid=1, entry_tid=12818]]], state=connected] 2024-08-31T22:41:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_3.status 2024-08-31T22:41:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:41:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_3 2024-08-31T22:41:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:41:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:41:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_3 2024-08-31T22:41:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_3 '..*' 2024-08-31T22:41:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:41:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:41:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_3 2024-08-31T22:41:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:41:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-08-31T22:41:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_3.status 2024-08-31T22:41:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:41:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_3.status 2024-08-31T22:41:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-08-31T22:41:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_3.status 2024-08-31T22:41:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:41:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:41:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:41:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12819], [object_number=14, tag_tid=1, entry_tid=12818]]], state=connected] 2024-08-31T22:41:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12819], [object_number=14, tag_tid=1, entry_tid=12818]]], state=connected] 2024-08-31T22:41:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_3.status 2024-08-31T22:41:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:41:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:41:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-08-31T22:41:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:41:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:41:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:41:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:41:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:41:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-08-31T22:41:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_3': 2024-08-31T22:41:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:41:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:41:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:41:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a829f032c8 2024-08-31T22:41:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a829f032c8 2024-08-31T22:41:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:41:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:41:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:41:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:41:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:32:42 2024 2024-08-31T22:41:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:32:42 2024 2024-08-31T22:41:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:32:42 2024 2024-08-31T22:41:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:41:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a829f032c8 2024-08-31T22:41:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:41:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:41:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 0f79ddcb-efe8-4cf3-9a5f-61d3edafc267 2024-08-31T22:41:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:41:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_3 snap 2024-08-31T22:41:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:41:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:41:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:41:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_3.export 2024-08-31T22:41:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_3.export 2024-08-31T22:41:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_3.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_3.export 2024-08-31T22:41:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_3.export 2024-08-31T22:41:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:41:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_3.export 2024-08-31T22:41:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:41:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_3.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_3.export 2024-08-31T22:41:50.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_3.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_3.export 2024-08-31T22:41:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:41:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_4 2024-08-31T22:41:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_4 snap 2024-08-31T22:41:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:41:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:41:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:41:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:41:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-08-31T22:41:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:41:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-08-31T22:41:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:41:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:41:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:41:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:41:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:41:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-08-31T22:41:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:41:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:41:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:41:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:41:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-08-31T22:41:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:41:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:41:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:41:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:41:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:41:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:41:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:41:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:41:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:41:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:41:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:41:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-08-31T22:41:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:41:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:41:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:41:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:41:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:41:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:41:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:41:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:41:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:41:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:41:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:41:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:41:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:41:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:41:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:41:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:41:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:41:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:41:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:41:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:41:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:41:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:41:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:41:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:41:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:41:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:41:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:41:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-08-31T22:41:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:no valid command found; 5 closest matches: 2024-08-31T22:41:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status mirror/image_41 2024-08-31T22:41:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status mirror/image_43 2024-08-31T22:41:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status mirror/image_45 2024-08-31T22:41:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status mirror/image_47 2024-08-31T22:41:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status mirror/image_49 2024-08-31T22:41:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:41:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:41:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-08-31T22:41:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:41:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:41:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:41:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:41:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:41:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:41:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:41:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:41:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:41:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:41:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:41:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:41:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:41:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:41:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:41:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:41:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:41:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:41:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:41:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:41:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:41:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:41:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:41:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:41:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:41:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:41:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:41:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:41:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:41:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:41:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:41:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:41:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-08-31T22:41:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:41:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:41:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:41:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:41:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:41:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:41:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:41:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-08-31T22:41:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:41:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:41:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:41:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:41:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_4 2024-08-31T22:41:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:41:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:41:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:41:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:41:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:41:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:41:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:41:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:41:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_4 2024-08-31T22:41:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:41:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:41:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_4 ']' 2024-08-31T22:41:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_4' 2024-08-31T22:41:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_4 2024-08-31T22:41:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:41:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:41:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:41:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:41:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:41:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:41:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:41:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:41:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:41:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_4 2024-08-31T22:41:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:41:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:41:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:41:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:41:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:41:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:41:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:41:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:41:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:41:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:41:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:41:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:41:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:41:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:41:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:41:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:41:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:41:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:41:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:41:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:41:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:41:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:41:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:41:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:41:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:41:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:41:54.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_4 2024-08-31T22:41:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:no valid command found; 5 closest matches: 2024-08-31T22:41:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush mirror/image_41 2024-08-31T22:41:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush mirror/image_43 2024-08-31T22:41:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush mirror/image_45 2024-08-31T22:41:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush mirror/image_47 2024-08-31T22:41:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush mirror/image_49 2024-08-31T22:41:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:41:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:41:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_4 2024-08-31T22:41:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:41:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:41:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:41:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:41:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:41:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:41:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:41:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:41:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:41:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:41:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:41:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:41:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:41:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:41:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:41:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:41:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:41:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:41:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:41:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:41:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:41:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:41:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:41:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:41:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:41:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:41:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:41:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:41:55.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:41:55.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:41:55.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:41:55.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_4 2024-08-31T22:41:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_4 2024-08-31T22:41:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:41:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:41:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_4 2024-08-31T22:41:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_4 '' 2024-08-31T22:41:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:41:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:41:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_4 2024-08-31T22:41:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:41:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-08-31T22:41:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_4.status 2024-08-31T22:41:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:41:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_4.status 2024-08-31T22:41:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-08-31T22:41:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_4.status 2024-08-31T22:41:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:41:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:41:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:41:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10051], [object_number=10, tag_tid=1, entry_tid=10050]]], state=connected] 2024-08-31T22:41:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10051], [object_number=10, tag_tid=1, entry_tid=10050]]], state=connected] 2024-08-31T22:41:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_4.status 2024-08-31T22:41:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:41:55.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_4 2024-08-31T22:41:55.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:41:55.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:41:55.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_4 2024-08-31T22:41:55.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_4 '..*' 2024-08-31T22:41:55.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:41:55.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:41:55.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_4 2024-08-31T22:41:55.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:41:55.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-08-31T22:41:55.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_4.status 2024-08-31T22:41:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:41:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_4.status 2024-08-31T22:41:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-08-31T22:41:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_4.status 2024-08-31T22:41:55.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:41:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:41:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:41:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10051], [object_number=10, tag_tid=1, entry_tid=10050]]], state=connected] 2024-08-31T22:41:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10051], [object_number=10, tag_tid=1, entry_tid=10050]]], state=connected] 2024-08-31T22:41:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_4.status 2024-08-31T22:41:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:41:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:41:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-08-31T22:41:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:41:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:41:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:41:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:41:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:41:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-08-31T22:41:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_4': 2024-08-31T22:41:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:41:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:41:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:41:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10ab832d9c05 2024-08-31T22:41:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10ab832d9c05 2024-08-31T22:41:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:41:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:41:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:41:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:41:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:32:55 2024 2024-08-31T22:41:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:32:55 2024 2024-08-31T22:41:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:32:55 2024 2024-08-31T22:41:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:41:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10ab832d9c05 2024-08-31T22:41:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:41:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:41:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 9b464c47-9795-4a19-bd60-31030064c0e0 2024-08-31T22:41:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:41:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_4 snap 2024-08-31T22:41:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:41:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:41:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:41:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_4.export 2024-08-31T22:41:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_4.export 2024-08-31T22:41:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_4.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_4.export 2024-08-31T22:41:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_4.export 2024-08-31T22:41:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:41:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_4.export 2024-08-31T22:41:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:41:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_4.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_4.export 2024-08-31T22:41:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_4.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_4.export 2024-08-31T22:41:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:41:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_5 2024-08-31T22:41:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_5 snap 2024-08-31T22:41:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:41:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:41:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:41:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:41:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-08-31T22:41:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:41:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-08-31T22:41:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:41:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:41:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:41:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:41:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:41:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-08-31T22:41:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:41:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:41:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:41:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:41:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-08-31T22:41:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:41:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:41:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:41:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:41:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:41:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:41:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:41:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:41:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:41:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:41:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:41:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-08-31T22:41:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:41:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:41:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:41:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:41:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:41:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:41:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:41:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:41:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:41:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:41:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:41:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:41:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:41:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:41:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:41:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:41:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:41:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:41:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:41:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:41:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:41:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:41:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:41:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:41:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:41:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:41:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:41:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-08-31T22:41:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:41:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:41:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:41:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:41:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:41:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:41:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:41:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-08-31T22:41:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:41:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:41:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:41:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:41:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_5 2024-08-31T22:41:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:41:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:41:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:41:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:41:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:41:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:41:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:41:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:41:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_5 2024-08-31T22:41:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:41:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:41:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:41:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_5 ']' 2024-08-31T22:41:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_5' 2024-08-31T22:41:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_5 2024-08-31T22:41:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:41:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:41:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:41:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:41:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:41:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:41:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:41:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:41:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:41:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:41:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_5 2024-08-31T22:41:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:41:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:41:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:41:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:41:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:41:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:41:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:41:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:41:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:41:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:41:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:41:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:41:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:41:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:41:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:41:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:41:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:41:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:41:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:41:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:41:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:41:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:41:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:41:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:41:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:41:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:41:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:41:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:41:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:41:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_5 2024-08-31T22:41:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:41:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_5 2024-08-31T22:41:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:41:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:41:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_5 2024-08-31T22:41:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_5 '' 2024-08-31T22:41:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:41:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:41:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_5 2024-08-31T22:41:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:41:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-08-31T22:41:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_5.status 2024-08-31T22:41:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:41:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_5.status 2024-08-31T22:41:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-08-31T22:41:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_5.status 2024-08-31T22:42:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6503], [object_number=6, tag_tid=1, entry_tid=6502]]], state=connected] 2024-08-31T22:42:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6503], [object_number=6, tag_tid=1, entry_tid=6502]]], state=connected] 2024-08-31T22:42:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_5.status 2024-08-31T22:42:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_5 2024-08-31T22:42:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_5 2024-08-31T22:42:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_5 '..*' 2024-08-31T22:42:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_5 2024-08-31T22:42:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:42:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-08-31T22:42:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_5.status 2024-08-31T22:42:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_5.status 2024-08-31T22:42:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-08-31T22:42:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_5.status 2024-08-31T22:42:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6503], [object_number=6, tag_tid=1, entry_tid=6502]]], state=connected] 2024-08-31T22:42:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6503], [object_number=6, tag_tid=1, entry_tid=6502]]], state=connected] 2024-08-31T22:42:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_5.status 2024-08-31T22:42:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-08-31T22:42:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:42:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:42:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-08-31T22:42:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_5': 2024-08-31T22:42:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:42:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:42:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:42:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a8cb12478 2024-08-31T22:42:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a8cb12478 2024-08-31T22:42:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:42:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:42:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:42:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:42:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:33:05 2024 2024-08-31T22:42:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:33:05 2024 2024-08-31T22:42:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:34:09 2024 2024-08-31T22:42:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:42:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a8cb12478 2024-08-31T22:42:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:42:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:42:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: e267b985-b732-408a-b169-281246eca56b 2024-08-31T22:42:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:42:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_5 snap 2024-08-31T22:42:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:42:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:42:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_5.export 2024-08-31T22:42:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_5.export 2024-08-31T22:42:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_5.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_5.export 2024-08-31T22:42:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_5.export 2024-08-31T22:42:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_5.export 2024-08-31T22:42:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_5.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_5.export 2024-08-31T22:42:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_5.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_5.export 2024-08-31T22:42:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:42:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_6 2024-08-31T22:42:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_6 snap 2024-08-31T22:42:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:42:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:42:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-08-31T22:42:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:42:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-08-31T22:42:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:42:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:42:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-08-31T22:42:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:42:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:42:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:42:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:42:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-08-31T22:42:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:42:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:42:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:42:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:42:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:42:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:42:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-08-31T22:42:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:42:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:42:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:42:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:42:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:42:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:42:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:42:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:42:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:42:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:42:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:42:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:42:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:42:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-08-31T22:42:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-08-31T22:42:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:42:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:42:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2024-08-31T22:42:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:42:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:42:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:42:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:42:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:42:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:42:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:42:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:42:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:42:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:42:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:42:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:42:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:42:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:42:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:42:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:42:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:42:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:42:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:42:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2024-08-31T22:42:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:42:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:42:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:42:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:42:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:42:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:42:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:42:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-08-31T22:42:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:42:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:42:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_6 2024-08-31T22:42:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:42:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:42:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:42:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:42:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:42:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:42:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_6 2024-08-31T22:42:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:42:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:42:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_6 ']' 2024-08-31T22:42:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_6' 2024-08-31T22:42:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_6 2024-08-31T22:42:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:42:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:42:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:42:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:42:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:42:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_6 2024-08-31T22:42:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:42:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:42:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:42:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:42:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:42:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_6 2024-08-31T22:42:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-08-31T22:42:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:42:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:42:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_6 2024-08-31T22:42:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:42:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:42:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:42:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:42:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:42:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:42:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:42:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:42:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:42:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:42:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:42:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_6 2024-08-31T22:42:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_6 2024-08-31T22:42:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_6 2024-08-31T22:42:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_6 '' 2024-08-31T22:42:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_6 2024-08-31T22:42:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:42:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-08-31T22:42:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_6.status 2024-08-31T22:42:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_6.status 2024-08-31T22:42:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-08-31T22:42:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_6.status 2024-08-31T22:42:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-08-31T22:42:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-08-31T22:42:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_6.status 2024-08-31T22:42:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_6 2024-08-31T22:42:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_6 2024-08-31T22:42:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_6 '..*' 2024-08-31T22:42:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_6 2024-08-31T22:42:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:42:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-08-31T22:42:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_6.status 2024-08-31T22:42:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_6.status 2024-08-31T22:42:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-08-31T22:42:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_6.status 2024-08-31T22:42:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-08-31T22:42:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-08-31T22:42:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_6.status 2024-08-31T22:42:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-08-31T22:42:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:42:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:42:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:05.397 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:42:05.395+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff16b60 2024-08-31T22:42:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-08-31T22:42:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_6': 2024-08-31T22:42:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:42:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:42:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:42:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10ab9d8ac1ee 2024-08-31T22:42:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10ab9d8ac1ee 2024-08-31T22:42:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:42:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:42:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:42:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:42:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:33:10 2024 2024-08-31T22:42:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:33:10 2024 2024-08-31T22:42:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:34:26 2024 2024-08-31T22:42:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:42:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10ab9d8ac1ee 2024-08-31T22:42:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:42:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:42:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 472b6592-1bd5-422d-975f-f158366f19c2 2024-08-31T22:42:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:42:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_6 snap 2024-08-31T22:42:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:42:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:42:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_6.export 2024-08-31T22:42:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_6.export 2024-08-31T22:42:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_6.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_6.export 2024-08-31T22:42:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_6.export 2024-08-31T22:42:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_6.export 2024-08-31T22:42:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_6.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_6.export 2024-08-31T22:42:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_6.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_6.export 2024-08-31T22:42:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:42:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_7 2024-08-31T22:42:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_7 snap 2024-08-31T22:42:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:42:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:42:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-08-31T22:42:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:42:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-08-31T22:42:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:42:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:42:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-08-31T22:42:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:42:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:42:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:42:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:42:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-08-31T22:42:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:42:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:42:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:42:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:42:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:42:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:42:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-08-31T22:42:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:42:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:42:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:42:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:42:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:42:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:42:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:42:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:42:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:42:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:42:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:42:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:42:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:42:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-08-31T22:42:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:42:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:42:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:42:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:42:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:42:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:42:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:42:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-08-31T22:42:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:42:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:42:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_7 2024-08-31T22:42:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:42:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:42:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:42:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:42:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:42:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:42:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_7 2024-08-31T22:42:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:42:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:42:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_7 ']' 2024-08-31T22:42:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_7' 2024-08-31T22:42:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_7 2024-08-31T22:42:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:42:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:42:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:42:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:42:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:42:08.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:08.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_7 2024-08-31T22:42:08.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:42:08.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:08.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:42:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:42:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:42:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:42:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:08.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:08.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:08.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:08.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_7 2024-08-31T22:42:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_7 2024-08-31T22:42:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_7 2024-08-31T22:42:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_7 '' 2024-08-31T22:42:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_7 2024-08-31T22:42:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:42:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-08-31T22:42:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_7.status 2024-08-31T22:42:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_7.status 2024-08-31T22:42:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-08-31T22:42:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_7.status 2024-08-31T22:42:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5327], [object_number=6, tag_tid=1, entry_tid=5326]]], state=connected] 2024-08-31T22:42:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5327], [object_number=6, tag_tid=1, entry_tid=5326]]], state=connected] 2024-08-31T22:42:08.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_7.status 2024-08-31T22:42:08.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_7 2024-08-31T22:42:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_7 2024-08-31T22:42:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_7 '..*' 2024-08-31T22:42:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_7 2024-08-31T22:42:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:42:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-08-31T22:42:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_7.status 2024-08-31T22:42:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_7.status 2024-08-31T22:42:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-08-31T22:42:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_7.status 2024-08-31T22:42:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5327], [object_number=6, tag_tid=1, entry_tid=5326]]], state=connected] 2024-08-31T22:42:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5327], [object_number=6, tag_tid=1, entry_tid=5326]]], state=connected] 2024-08-31T22:42:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_7.status 2024-08-31T22:42:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:42:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:42:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-08-31T22:42:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:42:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-08-31T22:42:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:42:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:42:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:42:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:42:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:42:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:42:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:42:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:42:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:42:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:42:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=1 2024-08-31T22:42:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 1 2024-08-31T22:42:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:42:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:42:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:42:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_7 2024-08-31T22:42:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:42:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:42:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_7 ']' 2024-08-31T22:42:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_7' 2024-08-31T22:42:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_7 2024-08-31T22:42:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:42:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:42:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:42:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:42:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:42:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_7 2024-08-31T22:42:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:42:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:42:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:42:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:42:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:42:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_7 2024-08-31T22:42:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_7 2024-08-31T22:42:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_7 2024-08-31T22:42:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_7 '' 2024-08-31T22:42:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_7 2024-08-31T22:42:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:42:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-08-31T22:42:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_7.status 2024-08-31T22:42:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_7.status 2024-08-31T22:42:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-08-31T22:42:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_7.status 2024-08-31T22:42:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5327], [object_number=6, tag_tid=1, entry_tid=5326]]], state=connected] 2024-08-31T22:42:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5327], [object_number=6, tag_tid=1, entry_tid=5326]]], state=connected] 2024-08-31T22:42:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_7.status 2024-08-31T22:42:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_7 2024-08-31T22:42:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_7 2024-08-31T22:42:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_7 '..*' 2024-08-31T22:42:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_7 2024-08-31T22:42:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:42:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-08-31T22:42:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_7.status 2024-08-31T22:42:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_7.status 2024-08-31T22:42:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-08-31T22:42:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_7.status 2024-08-31T22:42:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5327], [object_number=6, tag_tid=1, entry_tid=5326]]], state=connected] 2024-08-31T22:42:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5327], [object_number=6, tag_tid=1, entry_tid=5326]]], state=connected] 2024-08-31T22:42:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_7.status 2024-08-31T22:42:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-08-31T22:42:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:42:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:42:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-08-31T22:42:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_7': 2024-08-31T22:42:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:42:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:42:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:42:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a8306cd1e9 2024-08-31T22:42:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a8306cd1e9 2024-08-31T22:42:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:42:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:42:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:42:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:42:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:33:17 2024 2024-08-31T22:42:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:33:17 2024 2024-08-31T22:42:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:33:17 2024 2024-08-31T22:42:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:42:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a8306cd1e9 2024-08-31T22:42:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:42:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:42:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 10f605d6-a278-42e6-ae5a-fef1ccecff95 2024-08-31T22:42:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:42:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_7 snap 2024-08-31T22:42:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:42:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:42:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_7.export 2024-08-31T22:42:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_7.export 2024-08-31T22:42:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_7.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_7.export 2024-08-31T22:42:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_7.export 2024-08-31T22:42:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_7.export 2024-08-31T22:42:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_7.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_7.export 2024-08-31T22:42:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_7.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_7.export 2024-08-31T22:42:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:42:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_8 2024-08-31T22:42:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_8 snap 2024-08-31T22:42:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:42:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:42:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-08-31T22:42:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:42:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-08-31T22:42:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:42:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:42:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-08-31T22:42:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:42:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:42:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:42:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:42:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-08-31T22:42:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:42:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:42:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:42:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:42:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:42:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:42:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-08-31T22:42:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:42:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:42:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:42:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:42:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:42:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:42:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:42:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:42:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:42:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:42:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:42:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:42:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:42:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-08-31T22:42:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-08-31T22:42:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:42:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:42:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2024-08-31T22:42:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:42:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:42:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:42:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:42:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:42:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:42:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:42:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:42:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:42:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:42:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:42:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:42:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:42:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:42:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:42:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:42:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:42:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:42:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:42:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2024-08-31T22:42:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:42:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:42:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:42:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:42:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:42:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:42:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:42:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-08-31T22:42:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:42:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:42:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_8 2024-08-31T22:42:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:42:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:42:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:42:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:42:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:42:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:42:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_8 2024-08-31T22:42:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:42:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:42:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_8 ']' 2024-08-31T22:42:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_8' 2024-08-31T22:42:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_8 2024-08-31T22:42:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:42:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:42:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:42:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:42:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:42:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_8 2024-08-31T22:42:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:42:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:42:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:42:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:42:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:42:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_8 2024-08-31T22:42:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-08-31T22:42:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:42:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:42:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_8 2024-08-31T22:42:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:42:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:42:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:42:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:42:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:42:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:42:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:42:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:42:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:42:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:42:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:42:13.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_8 2024-08-31T22:42:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_8 2024-08-31T22:42:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_8 2024-08-31T22:42:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_8 '' 2024-08-31T22:42:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_8 2024-08-31T22:42:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:42:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-08-31T22:42:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_8.status 2024-08-31T22:42:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_8.status 2024-08-31T22:42:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-08-31T22:42:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_8.status 2024-08-31T22:42:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5559], [object_number=6, tag_tid=1, entry_tid=5558]]], state=connected] 2024-08-31T22:42:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5559], [object_number=6, tag_tid=1, entry_tid=5558]]], state=connected] 2024-08-31T22:42:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_8.status 2024-08-31T22:42:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:42:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_8 2024-08-31T22:42:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_8 2024-08-31T22:42:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_8 '..*' 2024-08-31T22:42:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_8 2024-08-31T22:42:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:42:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-08-31T22:42:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_8.status 2024-08-31T22:42:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_8.status 2024-08-31T22:42:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-08-31T22:42:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_8.status 2024-08-31T22:42:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5559], [object_number=6, tag_tid=1, entry_tid=5558]]], state=connected] 2024-08-31T22:42:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5559], [object_number=6, tag_tid=1, entry_tid=5558]]], state=connected] 2024-08-31T22:42:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_8.status 2024-08-31T22:42:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:42:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:42:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-08-31T22:42:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:42:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:42:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-08-31T22:42:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_8': 2024-08-31T22:42:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:42:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:42:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:42:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10ab99020d24 2024-08-31T22:42:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10ab99020d24 2024-08-31T22:42:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:42:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:42:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:42:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:42:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:33:23 2024 2024-08-31T22:42:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:33:23 2024 2024-08-31T22:42:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:33:23 2024 2024-08-31T22:42:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:42:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10ab99020d24 2024-08-31T22:42:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:42:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:42:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 0ad13246-a99d-40fe-8974-e61b1be9d8fb 2024-08-31T22:42:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:42:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_8 snap 2024-08-31T22:42:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:42:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:42:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_8.export 2024-08-31T22:42:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_8.export 2024-08-31T22:42:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_8.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_8.export 2024-08-31T22:42:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_8.export 2024-08-31T22:42:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_8.export 2024-08-31T22:42:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_8.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_8.export 2024-08-31T22:42:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_8.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_8.export 2024-08-31T22:42:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:42:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_9 2024-08-31T22:42:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_9 snap 2024-08-31T22:42:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:42:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:42:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-08-31T22:42:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:17.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:42:17.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-08-31T22:42:17.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:17.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:17.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:42:17.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:42:17.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:17.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:17.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-08-31T22:42:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:42:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:42:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:42:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:42:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-08-31T22:42:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:42:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:42:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:42:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:42:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:42:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:42:18.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:18.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-08-31T22:42:18.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:42:18.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:18.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:18.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:18.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:18.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:42:18.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:18.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:18.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:42:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:42:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:42:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:42:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:42:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:42:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:42:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:42:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:42:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:42:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:42:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-08-31T22:42:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:42:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:42:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:42:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:42:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:42:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:42:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:42:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-08-31T22:42:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:42:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:42:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_9 2024-08-31T22:42:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:42:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:42:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:42:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:42:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:42:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:42:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_9 2024-08-31T22:42:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:42:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:42:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_9 ']' 2024-08-31T22:42:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_9' 2024-08-31T22:42:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_9 2024-08-31T22:42:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:42:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:42:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:42:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:42:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:42:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_9 2024-08-31T22:42:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:42:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:42:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:42:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:42:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:42:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_9 2024-08-31T22:42:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_9 2024-08-31T22:42:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_9 2024-08-31T22:42:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_9 '' 2024-08-31T22:42:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_9 2024-08-31T22:42:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:42:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_9.status 2024-08-31T22:42:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_9.status 2024-08-31T22:42:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_9.status 2024-08-31T22:42:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2024-08-31T22:42:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_9.status 2024-08-31T22:42:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:42:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:42:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9943], [object_number=10, tag_tid=1, entry_tid=9942]]], state=connected] 2024-08-31T22:42:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9945], [object_number=8, tag_tid=1, entry_tid=9944], [object_number=11, tag_tid=1, entry_tid=9943], [object_number=10, tag_tid=1, entry_tid=9942]]], state=connected] 2024-08-31T22:42:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_9.status 2024-08-31T22:42:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:42:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_9 2024-08-31T22:42:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_9 2024-08-31T22:42:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_9 '..*' 2024-08-31T22:42:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_9 2024-08-31T22:42:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:42:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_9.status 2024-08-31T22:42:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_9.status 2024-08-31T22:42:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_9.status 2024-08-31T22:42:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2024-08-31T22:42:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_9.status 2024-08-31T22:42:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:42:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:42:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9943], [object_number=10, tag_tid=1, entry_tid=9942]]], state=connected] 2024-08-31T22:42:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9945], [object_number=8, tag_tid=1, entry_tid=9944], [object_number=11, tag_tid=1, entry_tid=9943], [object_number=10, tag_tid=1, entry_tid=9942]]], state=connected] 2024-08-31T22:42:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_9.status 2024-08-31T22:42:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9945' 2024-08-31T22:42:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9945' 2024-08-31T22:42:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9945' '!=' '' 2024-08-31T22:42:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:42:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9945' = '' 2024-08-31T22:42:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=9, tag_tid=1, entry_tid=9945' 2024-08-31T22:42:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:42:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:42:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=2 2024-08-31T22:42:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=9945' 2024-08-31T22:42:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:42:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=1 2024-08-31T22:42:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:42:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:42:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:18.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:42:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=9945' 2024-08-31T22:42:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:42:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=9945 2024-08-31T22:42:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 2 = 1 -a 1 -le 9945 2024-08-31T22:42:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:42:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:42:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:42:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_9 2024-08-31T22:42:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:42:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:42:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_9 ']' 2024-08-31T22:42:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_9' 2024-08-31T22:42:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_9 2024-08-31T22:42:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:42:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:42:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:42:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:42:19.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:19.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:42:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_9 2024-08-31T22:42:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:42:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:42:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:42:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:42:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:42:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_9 2024-08-31T22:42:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_9 2024-08-31T22:42:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_9 2024-08-31T22:42:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_9 '' 2024-08-31T22:42:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_9 2024-08-31T22:42:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:42:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_9.status 2024-08-31T22:42:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_9.status 2024-08-31T22:42:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_9.status 2024-08-31T22:42:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_9.status 2024-08-31T22:42:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2024-08-31T22:42:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:42:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:42:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9943], [object_number=10, tag_tid=1, entry_tid=9942]]], state=connected] 2024-08-31T22:42:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9943], [object_number=10, tag_tid=1, entry_tid=9942]]], state=connected] 2024-08-31T22:42:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_9.status 2024-08-31T22:42:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:42:19.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_9 2024-08-31T22:42:19.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:19.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:19.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_9 2024-08-31T22:42:19.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_9 '..*' 2024-08-31T22:42:19.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_9 2024-08-31T22:42:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:42:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_9.status 2024-08-31T22:42:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_9.status 2024-08-31T22:42:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_9.status 2024-08-31T22:42:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2024-08-31T22:42:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_9.status 2024-08-31T22:42:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:42:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:42:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9943], [object_number=10, tag_tid=1, entry_tid=9942]]], state=connected] 2024-08-31T22:42:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9943], [object_number=10, tag_tid=1, entry_tid=9942]]], state=connected] 2024-08-31T22:42:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_9.status 2024-08-31T22:42:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:42:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:42:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-08-31T22:42:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:42:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:42:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-08-31T22:42:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_9': 2024-08-31T22:42:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:42:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:42:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:42:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a8e4bc042d 2024-08-31T22:42:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a8e4bc042d 2024-08-31T22:42:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:42:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:42:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:42:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:42:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:33:29 2024 2024-08-31T22:42:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:33:29 2024 2024-08-31T22:42:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:33:29 2024 2024-08-31T22:42:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:42:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a8e4bc042d 2024-08-31T22:42:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:42:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:42:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: ffcc236b-c0d5-466c-abfd-4ea414e9702f 2024-08-31T22:42:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:42:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_9 snap 2024-08-31T22:42:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:42:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:42:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_9.export 2024-08-31T22:42:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_9.export 2024-08-31T22:42:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_9.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_9.export 2024-08-31T22:42:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_9.export 2024-08-31T22:42:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:20.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_9.export 2024-08-31T22:42:21.068 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_9.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_9.export 2024-08-31T22:42:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_9.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_9.export 2024-08-31T22:42:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:42:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_10 2024-08-31T22:42:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_10 snap 2024-08-31T22:42:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:42:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:42:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-08-31T22:42:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:42:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-08-31T22:42:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:42:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:42:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-08-31T22:42:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:42:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:42:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:42:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:42:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-08-31T22:42:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:42:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:42:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:42:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:42:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:42:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:42:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-08-31T22:42:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:42:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:42:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:42:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:42:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:42:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:42:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:42:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:42:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:42:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:42:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:42:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:42:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:42:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-08-31T22:42:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-08-31T22:42:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:42:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:42:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-08-31T22:42:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:42:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:42:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:42:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:42:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:42:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:42:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:42:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:42:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:42:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:42:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:42:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:42:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:42:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:42:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:42:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:42:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:42:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:42:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:42:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-08-31T22:42:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:42:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:42:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:42:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:42:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:42:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:42:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:42:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-08-31T22:42:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:42:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:42:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_10 2024-08-31T22:42:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:42:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:42:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:42:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:42:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:42:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:42:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_10 2024-08-31T22:42:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:42:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:42:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_10 ']' 2024-08-31T22:42:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-08-31T22:42:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-08-31T22:42:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:42:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:42:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:42:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:42:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:42:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-08-31T22:42:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:42:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:42:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:42:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:42:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:42:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:25.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-08-31T22:42:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-08-31T22:42:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:42:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:42:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-08-31T22:42:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:42:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:42:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:42:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:42:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:42:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:42:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:42:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:42:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:42:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:42:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:42:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-08-31T22:42:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-08-31T22:42:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_10 2024-08-31T22:42:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-08-31T22:42:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_10 2024-08-31T22:42:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:42:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-08-31T22:42:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_10.status 2024-08-31T22:42:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_10.status 2024-08-31T22:42:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_10.status 2024-08-31T22:42:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-08-31T22:42:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:42:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:42:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8263], [object_number=10, tag_tid=1, entry_tid=8262]]], state=connected] 2024-08-31T22:42:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8263], [object_number=10, tag_tid=1, entry_tid=8262]]], state=connected] 2024-08-31T22:42:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_10.status 2024-08-31T22:42:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:42:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-08-31T22:42:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_10 2024-08-31T22:42:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-08-31T22:42:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_10 2024-08-31T22:42:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:42:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-08-31T22:42:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_10.status 2024-08-31T22:42:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_10.status 2024-08-31T22:42:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-08-31T22:42:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_10.status 2024-08-31T22:42:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:42:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:42:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8263], [object_number=10, tag_tid=1, entry_tid=8262]]], state=connected] 2024-08-31T22:42:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8263], [object_number=10, tag_tid=1, entry_tid=8262]]], state=connected] 2024-08-31T22:42:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_10.status 2024-08-31T22:42:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:42:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:42:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-08-31T22:42:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:42:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:42:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-08-31T22:42:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_10': 2024-08-31T22:42:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:42:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:42:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:42:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10ab868ca049 2024-08-31T22:42:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10ab868ca049 2024-08-31T22:42:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:42:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:42:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:42:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:42:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:33:42 2024 2024-08-31T22:42:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:33:42 2024 2024-08-31T22:42:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:33:42 2024 2024-08-31T22:42:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:42:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10ab868ca049 2024-08-31T22:42:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:42:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:42:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 0eacebcd-2cd3-46ce-940e-3b2eb4a5b58b 2024-08-31T22:42:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_10 snap 2024-08-31T22:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_10.export 2024-08-31T22:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_10.export 2024-08-31T22:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_10.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_10.export 2024-08-31T22:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_10.export 2024-08-31T22:42:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_10.export 2024-08-31T22:42:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_10.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_10.export 2024-08-31T22:42:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_10.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_10.export 2024-08-31T22:42:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:42:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_11 2024-08-31T22:42:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_11 snap 2024-08-31T22:42:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_11 2024-08-31T22:42:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:42:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-08-31T22:42:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_11 2024-08-31T22:42:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-08-31T22:42:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_11 2024-08-31T22:42:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:42:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:29.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-08-31T22:42:29.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:29.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_11 2024-08-31T22:42:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:42:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:42:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:42:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-08-31T22:42:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:42:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:42:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:42:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:42:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:42:29.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:29.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:42:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-08-31T22:42:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:42:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:42:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:42:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:42:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:42:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:42:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:42:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:42:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:42:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:42:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:42:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:42:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:42:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-08-31T22:42:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:42:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:42:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:42:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:42:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:42:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:42:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:42:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-08-31T22:42:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_11 2024-08-31T22:42:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:42:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_11 2024-08-31T22:42:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_11 2024-08-31T22:42:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:42:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:42:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:42:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:42:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:42:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_11 2024-08-31T22:42:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_11 2024-08-31T22:42:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:42:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_11 ']' 2024-08-31T22:42:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_11' 2024-08-31T22:42:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_11 2024-08-31T22:42:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:42:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:42:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:42:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:42:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:42:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_11 2024-08-31T22:42:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:42:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:42:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:42:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:42:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:42:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_11 2024-08-31T22:42:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_11 2024-08-31T22:42:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_11 2024-08-31T22:42:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_11 '' 2024-08-31T22:42:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_11 2024-08-31T22:42:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:42:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-08-31T22:42:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_11.status 2024-08-31T22:42:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_11.status 2024-08-31T22:42:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-08-31T22:42:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_11.status 2024-08-31T22:42:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:42:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:42:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8627], [object_number=10, tag_tid=1, entry_tid=8626]]], state=connected] 2024-08-31T22:42:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=8627], [object_number=10, tag_tid=1, entry_tid=8626], [object_number=9, tag_tid=1, entry_tid=8625], [object_number=8, tag_tid=1, entry_tid=8624]]], state=connected] 2024-08-31T22:42:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_11.status 2024-08-31T22:42:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-08-31T22:42:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_11 2024-08-31T22:42:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_11 2024-08-31T22:42:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_11 '..*' 2024-08-31T22:42:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_11 2024-08-31T22:42:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:42:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-08-31T22:42:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_11.status 2024-08-31T22:42:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_11.status 2024-08-31T22:42:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-08-31T22:42:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_11.status 2024-08-31T22:42:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:42:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:42:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8627], [object_number=10, tag_tid=1, entry_tid=8626]]], state=connected] 2024-08-31T22:42:29.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=8627], [object_number=10, tag_tid=1, entry_tid=8626], [object_number=9, tag_tid=1, entry_tid=8625], [object_number=8, tag_tid=1, entry_tid=8624]]], state=connected] 2024-08-31T22:42:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_11.status 2024-08-31T22:42:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=11, tag_tid=1, entry_tid=8627' 2024-08-31T22:42:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=11, tag_tid=1, entry_tid=8627' 2024-08-31T22:42:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=8627' '!=' '' 2024-08-31T22:42:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:42:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=8627' = '' 2024-08-31T22:42:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=11, tag_tid=1, entry_tid=8627' 2024-08-31T22:42:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=9, tag_tid=3, entry_tid=1' 2024-08-31T22:42:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:42:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:42:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=11, tag_tid=1, entry_tid=8627' 2024-08-31T22:42:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:42:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=1 2024-08-31T22:42:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=9, tag_tid=3, entry_tid=1' 2024-08-31T22:42:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:42:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:29.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:42:29.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=11, tag_tid=1, entry_tid=8627' 2024-08-31T22:42:29.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:42:29.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=8627 2024-08-31T22:42:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 1 -a 1 -le 8627 2024-08-31T22:42:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:42:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:42:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:42:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_11 2024-08-31T22:42:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_11 2024-08-31T22:42:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:42:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_11 ']' 2024-08-31T22:42:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_11' 2024-08-31T22:42:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_11 2024-08-31T22:42:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:42:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:42:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:42:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:42:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:42:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_11 2024-08-31T22:42:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:42:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:42:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:42:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:42:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:42:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_11 2024-08-31T22:42:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_11 2024-08-31T22:42:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_11 2024-08-31T22:42:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_11 '' 2024-08-31T22:42:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_11 2024-08-31T22:42:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:42:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-08-31T22:42:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_11.status 2024-08-31T22:42:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_11.status 2024-08-31T22:42:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_11.status 2024-08-31T22:42:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-08-31T22:42:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:42:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:42:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8627], [object_number=10, tag_tid=1, entry_tid=8626]]], state=connected] 2024-08-31T22:42:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8627], [object_number=10, tag_tid=1, entry_tid=8626]]], state=connected] 2024-08-31T22:42:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_11.status 2024-08-31T22:42:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-08-31T22:42:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_11 2024-08-31T22:42:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_11 2024-08-31T22:42:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_11 '..*' 2024-08-31T22:42:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_11 2024-08-31T22:42:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:42:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-08-31T22:42:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_11.status 2024-08-31T22:42:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_11.status 2024-08-31T22:42:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-08-31T22:42:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_11.status 2024-08-31T22:42:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:42:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:42:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8627], [object_number=10, tag_tid=1, entry_tid=8626]]], state=connected] 2024-08-31T22:42:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8627], [object_number=10, tag_tid=1, entry_tid=8626]]], state=connected] 2024-08-31T22:42:30.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_11.status 2024-08-31T22:42:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-08-31T22:42:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-08-31T22:42:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-08-31T22:42:30.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:30.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:30.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_11 2024-08-31T22:42:30.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:30.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:30.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:42:30.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-08-31T22:42:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_11': 2024-08-31T22:42:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:42:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:42:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a88eda51f9 2024-08-31T22:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a88eda51f9 2024-08-31T22:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:33:56 2024 2024-08-31T22:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:33:56 2024 2024-08-31T22:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:34:57 2024 2024-08-31T22:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a88eda51f9 2024-08-31T22:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:42:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 59df3c86-0d77-4c82-9b97-8fd97d5645fb 2024-08-31T22:42:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:42:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_11 snap 2024-08-31T22:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_11 2024-08-31T22:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_11.export 2024-08-31T22:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_11.export 2024-08-31T22:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_11.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_11.export 2024-08-31T22:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_11.export 2024-08-31T22:42:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_11.export 2024-08-31T22:42:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_11.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_11.export 2024-08-31T22:42:32.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_11.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_11.export 2024-08-31T22:42:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:42:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_12 2024-08-31T22:42:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_12 snap 2024-08-31T22:42:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_12 2024-08-31T22:42:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:42:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-08-31T22:42:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_12 2024-08-31T22:42:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-08-31T22:42:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_12 2024-08-31T22:42:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:42:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-08-31T22:42:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_12 2024-08-31T22:42:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:42:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:42:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:42:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-08-31T22:42:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:42:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:42:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:42:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:42:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:42:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:42:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-08-31T22:42:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:42:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:42:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:42:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:42:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:42:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:42:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:42:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:42:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:42:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:42:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:42:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:42:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:42:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-08-31T22:42:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-08-31T22:42:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:42:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:42:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-08-31T22:42:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:42:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:42:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:42:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:42:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:42:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:42:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:42:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:42:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:42:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:42:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:42:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:42:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:42:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:42:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:42:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:42:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:42:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:42:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:42:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-08-31T22:42:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:42:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:42:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:42:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:42:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:42:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:42:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:42:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-08-31T22:42:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_12 2024-08-31T22:42:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:42:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_12 2024-08-31T22:42:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_12 2024-08-31T22:42:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:42:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:42:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:42:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:42:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:42:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_12 2024-08-31T22:42:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_12 2024-08-31T22:42:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:42:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_12 ']' 2024-08-31T22:42:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_12' 2024-08-31T22:42:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2024-08-31T22:42:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:42:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:42:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:42:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:42:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:42:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2024-08-31T22:42:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:42:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:42:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:42:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:42:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:42:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2024-08-31T22:42:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-08-31T22:42:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:42:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:42:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2024-08-31T22:42:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:42:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:42:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:42:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:42:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:42:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:42:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:42:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:42:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:42:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:42:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:42:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2024-08-31T22:42:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_12 2024-08-31T22:42:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_12 2024-08-31T22:42:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_12 '' 2024-08-31T22:42:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_12 2024-08-31T22:42:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:42:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-08-31T22:42:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_12.status 2024-08-31T22:42:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_12.status 2024-08-31T22:42:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-08-31T22:42:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_12.status 2024-08-31T22:42:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=3990], [object_number=7, tag_tid=1, entry_tid=3987]]], state=connected] 2024-08-31T22:42:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=3990], [object_number=7, tag_tid=1, entry_tid=3987]]], state=connected] 2024-08-31T22:42:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_12.status 2024-08-31T22:42:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2024-08-31T22:42:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_12 2024-08-31T22:42:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2024-08-31T22:42:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_12 2024-08-31T22:42:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:42:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-08-31T22:42:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_12.status 2024-08-31T22:42:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_12.status 2024-08-31T22:42:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-08-31T22:42:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_12.status 2024-08-31T22:42:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=3990], [object_number=7, tag_tid=1, entry_tid=3987]]], state=connected] 2024-08-31T22:42:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=3990], [object_number=7, tag_tid=1, entry_tid=3987]]], state=connected] 2024-08-31T22:42:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_12.status 2024-08-31T22:42:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-08-31T22:42:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_12 2024-08-31T22:42:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:42:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-08-31T22:42:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_12': 2024-08-31T22:42:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:42:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:42:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:42:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10ab19fc0703 2024-08-31T22:42:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10ab19fc0703 2024-08-31T22:42:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:42:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:42:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:42:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:42:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:34:03 2024 2024-08-31T22:42:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:34:03 2024 2024-08-31T22:42:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:35:04 2024 2024-08-31T22:42:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:42:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10ab19fc0703 2024-08-31T22:42:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:42:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:42:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 06e09953-d7f0-4f13-a16c-4c9829fa0253 2024-08-31T22:42:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:42:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_12 snap 2024-08-31T22:42:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_12 2024-08-31T22:42:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:42:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_12.export 2024-08-31T22:42:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_12.export 2024-08-31T22:42:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_12.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_12.export 2024-08-31T22:42:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_12.export 2024-08-31T22:42:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_12.export 2024-08-31T22:42:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_12.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_12.export 2024-08-31T22:42:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_12.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_12.export 2024-08-31T22:42:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:42:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_13 2024-08-31T22:42:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_13 snap 2024-08-31T22:42:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_13 2024-08-31T22:42:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:42:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-08-31T22:42:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_13 2024-08-31T22:42:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-08-31T22:42:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_13 2024-08-31T22:42:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:42:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:39.803 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:42:39.794+0000 7ff273ffe640 -1 reset not still connected to 0x56193c12d520 2024-08-31T22:42:39.803 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:42:39.794+0000 7ff273ffe640 -1 reset not still connected to 0x56193d3808f0 2024-08-31T22:42:39.803 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:42:39.794+0000 7ff273ffe640 -1 reset not still connected to 0x56194f1f8b60 2024-08-31T22:42:39.803 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:42:39.794+0000 7ff273ffe640 -1 reset not still connected to 0x561950712270 2024-08-31T22:42:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-08-31T22:42:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_13 2024-08-31T22:42:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:42:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:42:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:42:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:42:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-08-31T22:42:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:42:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:42:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:42:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:42:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:42:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-08-31T22:42:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:42:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:42:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:42:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:42:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:42:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:42:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:42:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:42:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:42:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:42:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:42:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:42:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:42:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-08-31T22:42:40.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:42:40.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:42:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:42:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:42:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:42:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:42:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:42:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-08-31T22:42:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_13 2024-08-31T22:42:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:42:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_13 2024-08-31T22:42:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_13 2024-08-31T22:42:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:42:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:42:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:42:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:42:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:42:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_13 2024-08-31T22:42:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_13 2024-08-31T22:42:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:42:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_13 ']' 2024-08-31T22:42:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_13' 2024-08-31T22:42:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2024-08-31T22:42:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:42:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:42:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:42:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:42:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:42:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2024-08-31T22:42:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:42:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:42:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:42:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:42:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:42:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2024-08-31T22:42:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_13 2024-08-31T22:42:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_13 2024-08-31T22:42:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_13 '' 2024-08-31T22:42:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_13 2024-08-31T22:42:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:42:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-08-31T22:42:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_13.status 2024-08-31T22:42:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_13.status 2024-08-31T22:42:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-08-31T22:42:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_13.status 2024-08-31T22:42:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:42:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:42:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13995], [object_number=14, tag_tid=1, entry_tid=13994]]], state=connected] 2024-08-31T22:42:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13995], [object_number=14, tag_tid=1, entry_tid=13994]]], state=connected] 2024-08-31T22:42:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_13.status 2024-08-31T22:42:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:42:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2024-08-31T22:42:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_13 2024-08-31T22:42:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2024-08-31T22:42:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_13 2024-08-31T22:42:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:42:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-08-31T22:42:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_13.status 2024-08-31T22:42:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_13.status 2024-08-31T22:42:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-08-31T22:42:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_13.status 2024-08-31T22:42:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:42:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:42:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13995], [object_number=14, tag_tid=1, entry_tid=13994]]], state=connected] 2024-08-31T22:42:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13995], [object_number=14, tag_tid=1, entry_tid=13994]]], state=connected] 2024-08-31T22:42:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_13.status 2024-08-31T22:42:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:42:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:42:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-08-31T22:42:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_13 2024-08-31T22:42:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:42:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-08-31T22:42:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_13': 2024-08-31T22:42:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:42:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:42:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:42:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a87193d63a 2024-08-31T22:42:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a87193d63a 2024-08-31T22:42:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:42:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:42:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:42:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:42:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:34:08 2024 2024-08-31T22:42:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:34:08 2024 2024-08-31T22:42:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:35:09 2024 2024-08-31T22:42:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:42:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a87193d63a 2024-08-31T22:42:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:42:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:42:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 83805a26-d50a-467f-be9d-63c3042a5afd 2024-08-31T22:42:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:42:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_13 snap 2024-08-31T22:42:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_13 2024-08-31T22:42:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:42:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_13.export 2024-08-31T22:42:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_13.export 2024-08-31T22:42:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_13.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_13.export 2024-08-31T22:42:42.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_13.export 2024-08-31T22:42:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:42.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_13.export 2024-08-31T22:42:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_13.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_13.export 2024-08-31T22:42:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_13.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_13.export 2024-08-31T22:42:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:42:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_14 2024-08-31T22:42:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_14 snap 2024-08-31T22:42:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_14 2024-08-31T22:42:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:42:44.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-08-31T22:42:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_14 2024-08-31T22:42:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-08-31T22:42:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_14 2024-08-31T22:42:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:42:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-08-31T22:42:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_14 2024-08-31T22:42:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:42:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:42:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:42:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-08-31T22:42:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:42:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:42:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:42:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:42:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:42:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:42:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-08-31T22:42:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:42:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:42:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:42:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:42:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:42:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:42:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:42:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:42:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:42:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:42:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:42:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:42:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:42:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-08-31T22:42:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-08-31T22:42:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:42:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:42:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-08-31T22:42:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:42:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:42:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:42:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:42:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:42:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:42:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:42:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:42:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:42:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:42:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:42:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:42:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:42:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:42:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:42:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:42:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:42:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:42:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:42:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-08-31T22:42:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:42:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:42:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:42:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:42:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:42:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:42:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:42:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-08-31T22:42:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_14 2024-08-31T22:42:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:42:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_14 2024-08-31T22:42:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_14 2024-08-31T22:42:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:42:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:42:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:42:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:42:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:42:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_14 2024-08-31T22:42:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_14 2024-08-31T22:42:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:42:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_14 ']' 2024-08-31T22:42:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_14' 2024-08-31T22:42:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2024-08-31T22:42:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:42:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:42:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:42:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:42:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:42:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2024-08-31T22:42:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:42:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:42:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:42:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:42:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:42:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2024-08-31T22:42:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-08-31T22:42:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:42:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:42:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2024-08-31T22:42:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:42:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:42:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:42:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:42:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:42:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:42:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:42:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:42:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:42:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:42:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:42:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2024-08-31T22:42:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_14 2024-08-31T22:42:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_14 2024-08-31T22:42:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_14 '' 2024-08-31T22:42:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_14 2024-08-31T22:42:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:42:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-08-31T22:42:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_14.status 2024-08-31T22:42:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_14.status 2024-08-31T22:42:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-08-31T22:42:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_14.status 2024-08-31T22:42:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4691], [object_number=6, tag_tid=1, entry_tid=4690]]], state=connected] 2024-08-31T22:42:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=4693], [object_number=4, tag_tid=1, entry_tid=4692], [object_number=7, tag_tid=1, entry_tid=4691], [object_number=6, tag_tid=1, entry_tid=4690]]], state=connected] 2024-08-31T22:42:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_14.status 2024-08-31T22:42:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:42:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2024-08-31T22:42:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_14 2024-08-31T22:42:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2024-08-31T22:42:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_14 2024-08-31T22:42:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:42:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-08-31T22:42:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_14.status 2024-08-31T22:42:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_14.status 2024-08-31T22:42:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-08-31T22:42:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_14.status 2024-08-31T22:42:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4691], [object_number=6, tag_tid=1, entry_tid=4690]]], state=connected] 2024-08-31T22:42:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=4693], [object_number=4, tag_tid=1, entry_tid=4692], [object_number=7, tag_tid=1, entry_tid=4691], [object_number=6, tag_tid=1, entry_tid=4690]]], state=connected] 2024-08-31T22:42:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_14.status 2024-08-31T22:42:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=4693' 2024-08-31T22:42:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=4693' 2024-08-31T22:42:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=4693' '!=' '' 2024-08-31T22:42:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:42:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=4693' = '' 2024-08-31T22:42:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=5, tag_tid=1, entry_tid=4693' 2024-08-31T22:42:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:42:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:42:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=2 2024-08-31T22:42:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=4693' 2024-08-31T22:42:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:42:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=1 2024-08-31T22:42:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:42:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:42:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:42:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=4693' 2024-08-31T22:42:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:42:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=4693 2024-08-31T22:42:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 2 = 1 -a 1 -le 4693 2024-08-31T22:42:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:42:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:42:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:42:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_14 2024-08-31T22:42:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_14 2024-08-31T22:42:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:42:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_14 ']' 2024-08-31T22:42:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_14' 2024-08-31T22:42:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2024-08-31T22:42:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:42:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:42:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:42:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:42:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:42:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2024-08-31T22:42:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:42:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:42:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:42:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:42:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:42:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2024-08-31T22:42:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-08-31T22:42:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:42:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:42:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2024-08-31T22:42:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:42:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:42:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:42:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:42:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:42:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:42:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:42:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:42:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:42:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:42:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:42:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2024-08-31T22:42:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_14 2024-08-31T22:42:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_14 2024-08-31T22:42:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_14 '' 2024-08-31T22:42:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_14 2024-08-31T22:42:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:42:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-08-31T22:42:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_14.status 2024-08-31T22:42:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_14.status 2024-08-31T22:42:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-08-31T22:42:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_14.status 2024-08-31T22:42:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4691], [object_number=6, tag_tid=1, entry_tid=4690]]], state=connected] 2024-08-31T22:42:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4691], [object_number=6, tag_tid=1, entry_tid=4690]]], state=connected] 2024-08-31T22:42:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_14.status 2024-08-31T22:42:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:42:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2024-08-31T22:42:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_14 2024-08-31T22:42:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2024-08-31T22:42:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_14 2024-08-31T22:42:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:42:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-08-31T22:42:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_14.status 2024-08-31T22:42:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_14.status 2024-08-31T22:42:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-08-31T22:42:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_14.status 2024-08-31T22:42:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4691], [object_number=6, tag_tid=1, entry_tid=4690]]], state=connected] 2024-08-31T22:42:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4691], [object_number=6, tag_tid=1, entry_tid=4690]]], state=connected] 2024-08-31T22:42:46.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_14.status 2024-08-31T22:42:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:42:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:42:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-08-31T22:42:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_14 2024-08-31T22:42:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:42:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-08-31T22:42:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_14': 2024-08-31T22:42:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:42:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:42:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:42:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10ab2e045e0 2024-08-31T22:42:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10ab2e045e0 2024-08-31T22:42:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:42:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:42:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:42:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:42:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:34:21 2024 2024-08-31T22:42:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:34:21 2024 2024-08-31T22:42:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:34:21 2024 2024-08-31T22:42:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:42:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10ab2e045e0 2024-08-31T22:42:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:42:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:42:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 8ec51a5a-6dcf-41d9-add3-7b26abf8ff2d 2024-08-31T22:42:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:42:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_14 snap 2024-08-31T22:42:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_14 2024-08-31T22:42:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:42:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_14.export 2024-08-31T22:42:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_14.export 2024-08-31T22:42:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_14.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_14.export 2024-08-31T22:42:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_14.export 2024-08-31T22:42:48.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_14.export 2024-08-31T22:42:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_14.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_14.export 2024-08-31T22:42:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_14.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_14.export 2024-08-31T22:42:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:42:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_15 2024-08-31T22:42:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_15 snap 2024-08-31T22:42:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_15 2024-08-31T22:42:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:42:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-08-31T22:42:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_15 2024-08-31T22:42:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-08-31T22:42:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_15 2024-08-31T22:42:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:42:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-08-31T22:42:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_15 2024-08-31T22:42:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:42:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:42:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:42:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-08-31T22:42:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:42:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:42:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:42:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:42:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:42:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:42:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-08-31T22:42:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:42:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:42:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:42:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:51.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:51.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:51.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:51.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:42:51.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:42:51.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:42:51.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:51.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:42:51.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:42:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:42:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:42:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:42:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:42:51.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:42:51.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-08-31T22:42:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:42:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:42:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:42:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:42:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:42:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:42:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:42:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-08-31T22:42:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_15 2024-08-31T22:42:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:42:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_15 2024-08-31T22:42:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_15 2024-08-31T22:42:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:42:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:42:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:42:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:42:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:42:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_15 2024-08-31T22:42:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_15 2024-08-31T22:42:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:42:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_15 ']' 2024-08-31T22:42:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_15' 2024-08-31T22:42:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_15 2024-08-31T22:42:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:42:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:42:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:42:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:42:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:42:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_15 2024-08-31T22:42:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:42:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:42:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:42:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:42:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:42:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_15 2024-08-31T22:42:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_15 2024-08-31T22:42:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_15 2024-08-31T22:42:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_15 '' 2024-08-31T22:42:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_15 2024-08-31T22:42:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:42:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-08-31T22:42:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_15.status 2024-08-31T22:42:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_15.status 2024-08-31T22:42:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-08-31T22:42:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_15.status 2024-08-31T22:42:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:42:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:42:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9966], [object_number=11, tag_tid=1, entry_tid=9963]]], state=connected] 2024-08-31T22:42:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9966], [object_number=11, tag_tid=1, entry_tid=9963]]], state=connected] 2024-08-31T22:42:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_15.status 2024-08-31T22:42:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:42:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_15 2024-08-31T22:42:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_15 2024-08-31T22:42:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_15 '..*' 2024-08-31T22:42:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_15 2024-08-31T22:42:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:42:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-08-31T22:42:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_15.status 2024-08-31T22:42:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:52.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_15.status 2024-08-31T22:42:52.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-08-31T22:42:52.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_15.status 2024-08-31T22:42:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:42:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:42:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9966], [object_number=11, tag_tid=1, entry_tid=9963]]], state=connected] 2024-08-31T22:42:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9966], [object_number=11, tag_tid=1, entry_tid=9963]]], state=connected] 2024-08-31T22:42:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_15.status 2024-08-31T22:42:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:42:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:42:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-08-31T22:42:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_15 2024-08-31T22:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-08-31T22:42:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_15': 2024-08-31T22:42:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:42:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:42:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:42:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a8fde270a5 2024-08-31T22:42:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a8fde270a5 2024-08-31T22:42:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:42:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:42:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:42:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:42:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:34:28 2024 2024-08-31T22:42:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:34:28 2024 2024-08-31T22:42:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:35:29 2024 2024-08-31T22:42:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:42:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a8fde270a5 2024-08-31T22:42:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:42:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:42:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 4253c17a-c571-43a2-957a-a8fc0262f832 2024-08-31T22:42:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:42:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_15 snap 2024-08-31T22:42:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_15 2024-08-31T22:42:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:42:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_15.export 2024-08-31T22:42:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_15.export 2024-08-31T22:42:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_15.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_15.export 2024-08-31T22:42:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_15.export 2024-08-31T22:42:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_15.export 2024-08-31T22:42:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_15.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_15.export 2024-08-31T22:42:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_15.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_15.export 2024-08-31T22:42:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:42:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_16 2024-08-31T22:42:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_16 snap 2024-08-31T22:42:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_16 2024-08-31T22:42:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:42:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-08-31T22:42:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_16 2024-08-31T22:42:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-08-31T22:42:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_16 2024-08-31T22:42:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:42:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-08-31T22:42:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_16 2024-08-31T22:42:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:42:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:42:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:42:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-08-31T22:42:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:42:55.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:42:55.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:55.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:55.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:42:55.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:42:55.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:55.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:42:55.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:42:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-08-31T22:42:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:42:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:42:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:42:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:42:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:42:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:42:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:42:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:42:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:42:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:42:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:42:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:42:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:42:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-08-31T22:42:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-08-31T22:42:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:42:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:42:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-08-31T22:42:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:42:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:42:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:42:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:42:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:42:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:42:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:42:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:42:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:42:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:42:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:42:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:42:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:42:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:42:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:42:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:42:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:42:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:42:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:42:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:42:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:42:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-08-31T22:42:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:42:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:42:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:42:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:42:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:42:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:42:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:42:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-08-31T22:42:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_16 2024-08-31T22:42:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:42:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_16 2024-08-31T22:42:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:42:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_16 2024-08-31T22:42:55.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:42:55.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:42:55.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:42:55.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:42:55.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:42:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_16 2024-08-31T22:42:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_16 2024-08-31T22:42:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:42:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_16 ']' 2024-08-31T22:42:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_16' 2024-08-31T22:42:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2024-08-31T22:42:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:42:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:42:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:42:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:42:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:42:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2024-08-31T22:42:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:42:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:42:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:42:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:42:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:42:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2024-08-31T22:42:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-08-31T22:42:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:42:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:42:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2024-08-31T22:42:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:42:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:42:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:42:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:42:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:42:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:42:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:42:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:42:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:42:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:42:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:42:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2024-08-31T22:42:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_16 2024-08-31T22:42:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_16 2024-08-31T22:42:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_16 '' 2024-08-31T22:42:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_16 2024-08-31T22:42:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:42:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-08-31T22:42:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_16.status 2024-08-31T22:42:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_16.status 2024-08-31T22:42:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-08-31T22:42:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_16.status 2024-08-31T22:42:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4590], [object_number=7, tag_tid=1, entry_tid=4587]]], state=connected] 2024-08-31T22:42:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4590], [object_number=7, tag_tid=1, entry_tid=4587]]], state=connected] 2024-08-31T22:42:56.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_16.status 2024-08-31T22:42:56.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2024-08-31T22:42:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_16 2024-08-31T22:42:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2024-08-31T22:42:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_16 2024-08-31T22:42:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:42:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-08-31T22:42:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_16.status 2024-08-31T22:42:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_16.status 2024-08-31T22:42:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-08-31T22:42:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_16.status 2024-08-31T22:42:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4590], [object_number=7, tag_tid=1, entry_tid=4587]]], state=connected] 2024-08-31T22:42:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4590], [object_number=7, tag_tid=1, entry_tid=4587]]], state=connected] 2024-08-31T22:42:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_16.status 2024-08-31T22:42:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:42:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:42:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-08-31T22:42:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:42:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-08-31T22:42:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:42:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:42:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:56.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:42:56.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:42:56.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:42:56.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:56.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:42:56.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:56.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:42:56.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:56.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:42:56.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:42:56.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:42:56.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:42:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=1 2024-08-31T22:42:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 1 2024-08-31T22:42:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:42:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:42:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:42:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_16 2024-08-31T22:42:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_16 2024-08-31T22:42:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:42:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_16 ']' 2024-08-31T22:42:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_16' 2024-08-31T22:42:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2024-08-31T22:42:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:42:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:42:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:42:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:42:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:42:56.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:42:56.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:42:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2024-08-31T22:42:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:42:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:42:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:42:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:42:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:42:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:42:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:42:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:42:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:42:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:42:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:42:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:42:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:42:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:42:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:42:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2024-08-31T22:42:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-08-31T22:42:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:42:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:42:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:42:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2024-08-31T22:42:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:42:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:42:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:42:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:42:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:42:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:42:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:42:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:42:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:42:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:42:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:42:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:42:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:42:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:42:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:42:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:42:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:42:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:42:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:42:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:42:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:42:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:42:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:42:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:42:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:42:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:42:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:42:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:42:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2024-08-31T22:42:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_16 2024-08-31T22:42:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_16 2024-08-31T22:42:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_16 '' 2024-08-31T22:42:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_16 2024-08-31T22:42:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:42:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-08-31T22:42:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_16.status 2024-08-31T22:42:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_16.status 2024-08-31T22:42:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-08-31T22:42:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_16.status 2024-08-31T22:42:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4590], [object_number=7, tag_tid=1, entry_tid=4587]]], state=connected] 2024-08-31T22:42:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4590], [object_number=7, tag_tid=1, entry_tid=4587]]], state=connected] 2024-08-31T22:42:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_16.status 2024-08-31T22:42:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2024-08-31T22:42:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_16 2024-08-31T22:42:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2024-08-31T22:42:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:42:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:42:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_16 2024-08-31T22:42:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:42:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-08-31T22:42:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_16.status 2024-08-31T22:42:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:42:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_16.status 2024-08-31T22:42:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-08-31T22:42:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_16.status 2024-08-31T22:42:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:42:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:42:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:42:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4590], [object_number=7, tag_tid=1, entry_tid=4587]]], state=connected] 2024-08-31T22:42:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4590], [object_number=7, tag_tid=1, entry_tid=4587]]], state=connected] 2024-08-31T22:42:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_16.status 2024-08-31T22:42:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:42:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-08-31T22:42:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:42:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_16 2024-08-31T22:42:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:42:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:42:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:42:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-08-31T22:42:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_16': 2024-08-31T22:42:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:42:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:42:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:42:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10aba38fd231 2024-08-31T22:42:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10aba38fd231 2024-08-31T22:42:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:42:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:42:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:42:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:42:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:34:40 2024 2024-08-31T22:42:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:34:40 2024 2024-08-31T22:42:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:35:40 2024 2024-08-31T22:42:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:42:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10aba38fd231 2024-08-31T22:42:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:42:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:42:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 534fb173-df21-4869-a37d-01d229fe43a4 2024-08-31T22:42:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:42:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_16 snap 2024-08-31T22:42:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_16 2024-08-31T22:42:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:42:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_16.export 2024-08-31T22:42:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_16.export 2024-08-31T22:42:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_16.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_16.export 2024-08-31T22:42:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_16.export 2024-08-31T22:42:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:58.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_16.export 2024-08-31T22:42:58.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:42:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_16.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_16.export 2024-08-31T22:42:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_16.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_16.export 2024-08-31T22:42:58.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:42:58.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:42:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_17 2024-08-31T22:42:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_17 snap 2024-08-31T22:42:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:42:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:42:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_17 2024-08-31T22:42:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:42:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:43:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-08-31T22:43:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_17 2024-08-31T22:43:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-08-31T22:43:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_17 2024-08-31T22:43:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:43:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-08-31T22:43:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_17 2024-08-31T22:43:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:43:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:43:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:43:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-08-31T22:43:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:43:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:43:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:43:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:43:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:43:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:43:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-08-31T22:43:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:43:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:43:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:43:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:02.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:43:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:43:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:43:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:43:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:43:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:43:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:43:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:43:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:43:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:43:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-08-31T22:43:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:43:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:43:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:43:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:43:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:43:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:43:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:43:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-08-31T22:43:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_17 2024-08-31T22:43:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:43:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_17 2024-08-31T22:43:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_17 2024-08-31T22:43:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:43:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:43:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:43:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:43:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_17 2024-08-31T22:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_17 2024-08-31T22:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_17 ']' 2024-08-31T22:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_17' 2024-08-31T22:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_17 2024-08-31T22:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:43:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:43:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:43:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:43:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:02.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_17 2024-08-31T22:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:43:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:43:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_17 2024-08-31T22:43:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_17 2024-08-31T22:43:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_17 2024-08-31T22:43:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_17 '' 2024-08-31T22:43:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_17 2024-08-31T22:43:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:43:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-08-31T22:43:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_17.status 2024-08-31T22:43:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_17.status 2024-08-31T22:43:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-08-31T22:43:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_17.status 2024-08-31T22:43:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:43:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:43:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7007], [object_number=10, tag_tid=1, entry_tid=7006]]], state=connected] 2024-08-31T22:43:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7007], [object_number=10, tag_tid=1, entry_tid=7006]]], state=connected] 2024-08-31T22:43:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_17.status 2024-08-31T22:43:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:43:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_17 2024-08-31T22:43:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_17 2024-08-31T22:43:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_17 '..*' 2024-08-31T22:43:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_17 2024-08-31T22:43:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:43:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-08-31T22:43:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_17.status 2024-08-31T22:43:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_17.status 2024-08-31T22:43:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-08-31T22:43:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_17.status 2024-08-31T22:43:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:43:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:43:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7007], [object_number=10, tag_tid=1, entry_tid=7006]]], state=connected] 2024-08-31T22:43:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7007], [object_number=10, tag_tid=1, entry_tid=7006]]], state=connected] 2024-08-31T22:43:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_17.status 2024-08-31T22:43:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:43:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:43:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-08-31T22:43:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_17 2024-08-31T22:43:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:43:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-08-31T22:43:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_17': 2024-08-31T22:43:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:43:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:43:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:43:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a89cd4cd49 2024-08-31T22:43:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a89cd4cd49 2024-08-31T22:43:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:43:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:43:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:43:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:43:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:34:45 2024 2024-08-31T22:43:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:34:45 2024 2024-08-31T22:43:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:34:45 2024 2024-08-31T22:43:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:43:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a89cd4cd49 2024-08-31T22:43:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:43:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:43:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 616d7e08-1a46-4f1b-866b-231d48e795a7 2024-08-31T22:43:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:43:04.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:04.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_17 snap 2024-08-31T22:43:04.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:04.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_17 2024-08-31T22:43:04.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:04.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:43:04.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_17.export 2024-08-31T22:43:04.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_17.export 2024-08-31T22:43:04.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_17.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_17.export 2024-08-31T22:43:04.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_17.export 2024-08-31T22:43:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_17.export 2024-08-31T22:43:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_17.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_17.export 2024-08-31T22:43:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_17.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_17.export 2024-08-31T22:43:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:43:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_18 2024-08-31T22:43:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_18 snap 2024-08-31T22:43:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_18 2024-08-31T22:43:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:43:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-08-31T22:43:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_18 2024-08-31T22:43:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-08-31T22:43:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_18 2024-08-31T22:43:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:43:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-08-31T22:43:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_18 2024-08-31T22:43:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:43:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:43:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:43:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-08-31T22:43:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:43:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:43:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:43:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:43:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:43:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:43:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-08-31T22:43:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:43:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:43:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:43:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:43:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:43:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:43:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:43:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:43:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:43:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:43:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:43:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:43:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:43:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-08-31T22:43:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-08-31T22:43:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:43:06.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:43:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-08-31T22:43:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:43:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:43:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:43:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:43:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:43:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:43:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:43:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:43:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:43:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:43:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:43:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:43:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:43:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:43:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:43:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:43:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:43:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:43:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:43:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:43:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:43:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:43:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-08-31T22:43:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:43:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:43:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:43:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:43:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:43:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:43:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:43:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-08-31T22:43:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_18 2024-08-31T22:43:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:43:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_18 2024-08-31T22:43:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_18 2024-08-31T22:43:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:43:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:43:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:43:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:43:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:43:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_18 2024-08-31T22:43:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_18 2024-08-31T22:43:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:43:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_18 ']' 2024-08-31T22:43:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_18' 2024-08-31T22:43:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_18 2024-08-31T22:43:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:43:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:43:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:43:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:43:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:43:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_18 2024-08-31T22:43:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:43:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:43:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:43:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:43:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:43:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:43:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:43:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:43:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:43:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:43:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:43:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:43:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:43:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_18 2024-08-31T22:43:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-08-31T22:43:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:43:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:43:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_18 2024-08-31T22:43:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:43:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:43:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:43:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:43:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:43:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:43:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:43:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:43:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:43:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:43:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:43:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:43:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:43:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:43:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:43:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:43:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:43:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:43:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:43:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:43:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:43:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:43:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_18 2024-08-31T22:43:07.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:07.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_18 2024-08-31T22:43:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_18 2024-08-31T22:43:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_18 '' 2024-08-31T22:43:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_18 2024-08-31T22:43:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:43:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-08-31T22:43:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_18.status 2024-08-31T22:43:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:07.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_18.status 2024-08-31T22:43:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_18.status 2024-08-31T22:43:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-08-31T22:43:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:43:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:43:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5603], [object_number=6, tag_tid=1, entry_tid=5602]]], state=connected] 2024-08-31T22:43:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5603], [object_number=6, tag_tid=1, entry_tid=5602]]], state=connected] 2024-08-31T22:43:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_18.status 2024-08-31T22:43:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:43:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_18 2024-08-31T22:43:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_18 2024-08-31T22:43:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_18 '..*' 2024-08-31T22:43:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_18 2024-08-31T22:43:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:43:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-08-31T22:43:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_18.status 2024-08-31T22:43:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:07.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_18.status 2024-08-31T22:43:07.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-08-31T22:43:07.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_18.status 2024-08-31T22:43:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:43:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:43:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5603], [object_number=6, tag_tid=1, entry_tid=5602]]], state=connected] 2024-08-31T22:43:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5603], [object_number=6, tag_tid=1, entry_tid=5602]]], state=connected] 2024-08-31T22:43:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_18.status 2024-08-31T22:43:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:43:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:43:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-08-31T22:43:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:43:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-08-31T22:43:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:43:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:43:07.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:43:07.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:43:07.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:43:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:43:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:43:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:43:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=2 2024-08-31T22:43:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:43:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:43:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:43:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:43:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:43:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:43:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:43:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=1 2024-08-31T22:43:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 2 -a 1 -le 1 2024-08-31T22:43:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:43:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:43:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:43:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_18 2024-08-31T22:43:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_18 2024-08-31T22:43:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:43:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_18 ']' 2024-08-31T22:43:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_18' 2024-08-31T22:43:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_18 2024-08-31T22:43:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:43:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:43:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:43:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:43:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:43:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_18 2024-08-31T22:43:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:43:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:43:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:43:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:43:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:43:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:43:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:43:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:43:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:43:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:43:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:43:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:43:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:43:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:07.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_18 2024-08-31T22:43:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-08-31T22:43:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:43:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:43:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_18 2024-08-31T22:43:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:43:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:43:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:43:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:43:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:43:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:43:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:43:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:43:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:43:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:43:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:43:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:43:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:43:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:43:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:43:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:43:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:43:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:43:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:43:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:43:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:43:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:43:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_18 2024-08-31T22:43:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_18 2024-08-31T22:43:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_18 2024-08-31T22:43:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_18 '' 2024-08-31T22:43:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_18 2024-08-31T22:43:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:43:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-08-31T22:43:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_18.status 2024-08-31T22:43:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_18.status 2024-08-31T22:43:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_18.status 2024-08-31T22:43:07.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-08-31T22:43:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:43:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:43:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5603], [object_number=6, tag_tid=1, entry_tid=5602]]], state=connected] 2024-08-31T22:43:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5603], [object_number=6, tag_tid=1, entry_tid=5602]]], state=connected] 2024-08-31T22:43:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_18.status 2024-08-31T22:43:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:43:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_18 2024-08-31T22:43:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_18 2024-08-31T22:43:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_18 '..*' 2024-08-31T22:43:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_18 2024-08-31T22:43:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:43:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-08-31T22:43:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_18.status 2024-08-31T22:43:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:07.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_18.status 2024-08-31T22:43:07.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-08-31T22:43:07.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_18.status 2024-08-31T22:43:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:43:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:43:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5603], [object_number=6, tag_tid=1, entry_tid=5602]]], state=connected] 2024-08-31T22:43:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5603], [object_number=6, tag_tid=1, entry_tid=5602]]], state=connected] 2024-08-31T22:43:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_18.status 2024-08-31T22:43:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:43:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:43:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-08-31T22:43:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_18 2024-08-31T22:43:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:43:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-08-31T22:43:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_18': 2024-08-31T22:43:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:43:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:43:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:43:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10ab4b124556 2024-08-31T22:43:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10ab4b124556 2024-08-31T22:43:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:43:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:43:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:43:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:43:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:34:55 2024 2024-08-31T22:43:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:34:55 2024 2024-08-31T22:43:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:36:00 2024 2024-08-31T22:43:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:43:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10ab4b124556 2024-08-31T22:43:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:43:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:43:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: c656f7a6-222b-4f70-af0d-b909d5dbc4b1 2024-08-31T22:43:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:43:09.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:09.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_18 snap 2024-08-31T22:43:09.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:09.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_18 2024-08-31T22:43:09.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:09.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:43:09.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_18.export 2024-08-31T22:43:09.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_18.export 2024-08-31T22:43:09.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_18.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_18.export 2024-08-31T22:43:09.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_18.export 2024-08-31T22:43:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_18.export 2024-08-31T22:43:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_18.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_18.export 2024-08-31T22:43:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_18.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_18.export 2024-08-31T22:43:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:43:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_19 2024-08-31T22:43:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_19 snap 2024-08-31T22:43:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_19 2024-08-31T22:43:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:43:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-08-31T22:43:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_19 2024-08-31T22:43:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-08-31T22:43:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_19 2024-08-31T22:43:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:43:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-08-31T22:43:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_19 2024-08-31T22:43:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:43:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:43:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:43:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-08-31T22:43:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:43:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:43:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:43:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:43:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:43:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:43:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-08-31T22:43:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:43:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:43:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:43:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:43:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:43:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:43:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:43:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:43:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:43:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:43:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:43:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:43:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:43:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-08-31T22:43:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:43:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:43:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:43:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:43:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:43:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:43:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:43:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-08-31T22:43:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_19 2024-08-31T22:43:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:43:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_19 2024-08-31T22:43:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_19 2024-08-31T22:43:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:43:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:43:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:43:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:43:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:43:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_19 2024-08-31T22:43:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_19 2024-08-31T22:43:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:43:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_19 ']' 2024-08-31T22:43:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_19' 2024-08-31T22:43:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_19 2024-08-31T22:43:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:43:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:43:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:43:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:43:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:43:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_19 2024-08-31T22:43:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:43:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:43:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:43:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:43:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:43:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:43:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:43:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:43:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:43:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:43:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:43:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:43:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:43:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_19 2024-08-31T22:43:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_19 2024-08-31T22:43:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_19 2024-08-31T22:43:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_19 '' 2024-08-31T22:43:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_19 2024-08-31T22:43:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:43:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-08-31T22:43:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_19.status 2024-08-31T22:43:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_19.status 2024-08-31T22:43:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-08-31T22:43:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_19.status 2024-08-31T22:43:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:43:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:43:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13035], [object_number=14, tag_tid=1, entry_tid=13034]]], state=connected] 2024-08-31T22:43:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13035], [object_number=14, tag_tid=1, entry_tid=13034]]], state=connected] 2024-08-31T22:43:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_19.status 2024-08-31T22:43:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:43:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_19 2024-08-31T22:43:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_19 2024-08-31T22:43:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_19 '..*' 2024-08-31T22:43:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_19 2024-08-31T22:43:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:43:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-08-31T22:43:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_19.status 2024-08-31T22:43:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_19.status 2024-08-31T22:43:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-08-31T22:43:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_19.status 2024-08-31T22:43:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:43:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:43:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13035], [object_number=14, tag_tid=1, entry_tid=13034]]], state=connected] 2024-08-31T22:43:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13035], [object_number=14, tag_tid=1, entry_tid=13034]]], state=connected] 2024-08-31T22:43:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_19.status 2024-08-31T22:43:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:43:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:43:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-08-31T22:43:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_19 2024-08-31T22:43:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:43:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-08-31T22:43:16.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_19': 2024-08-31T22:43:16.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:43:16.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:43:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:43:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a8dd48538 2024-08-31T22:43:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a8dd48538 2024-08-31T22:43:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:43:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:43:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:43:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:43:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:35:02 2024 2024-08-31T22:43:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:35:02 2024 2024-08-31T22:43:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:36:03 2024 2024-08-31T22:43:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:43:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a8dd48538 2024-08-31T22:43:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:43:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:43:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 0053e17e-e7b0-4d7a-a12d-130ee9f475ac 2024-08-31T22:43:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:43:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_19 snap 2024-08-31T22:43:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_19 2024-08-31T22:43:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:43:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_19.export 2024-08-31T22:43:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_19.export 2024-08-31T22:43:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_19.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_19.export 2024-08-31T22:43:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_19.export 2024-08-31T22:43:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_19.export 2024-08-31T22:43:17.177 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_19.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_19.export 2024-08-31T22:43:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_19.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_19.export 2024-08-31T22:43:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:43:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_20 2024-08-31T22:43:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_20 snap 2024-08-31T22:43:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_20 2024-08-31T22:43:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:43:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-08-31T22:43:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_20 2024-08-31T22:43:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-08-31T22:43:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_20 2024-08-31T22:43:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:43:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-08-31T22:43:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_20 2024-08-31T22:43:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:43:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:43:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:43:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-08-31T22:43:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:43:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:43:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:43:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:43:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:43:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:43:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-08-31T22:43:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:43:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:43:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:43:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:43:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:43:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:43:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:43:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:43:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:43:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:43:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:43:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:43:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:43:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-08-31T22:43:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-08-31T22:43:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:43:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:43:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-08-31T22:43:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:43:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:43:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:43:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:43:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:43:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:43:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:43:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:43:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:43:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:43:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:43:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:43:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:43:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:43:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:43:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:43:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:43:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:43:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:43:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:43:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:43:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:43:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-08-31T22:43:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:43:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:43:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:43:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:43:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:43:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:43:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:43:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-08-31T22:43:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_20 2024-08-31T22:43:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:43:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_20 2024-08-31T22:43:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_20 2024-08-31T22:43:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:43:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:43:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:43:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:43:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:43:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_20 2024-08-31T22:43:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_20 2024-08-31T22:43:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:43:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_20 ']' 2024-08-31T22:43:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_20' 2024-08-31T22:43:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2024-08-31T22:43:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:43:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:43:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:43:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:43:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:43:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2024-08-31T22:43:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:43:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:43:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:43:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:43:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:43:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:43:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:43:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:43:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:43:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:43:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:43:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:43:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:43:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2024-08-31T22:43:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-08-31T22:43:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:43:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:43:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2024-08-31T22:43:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:43:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:43:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:43:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:43:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:43:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:43:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:43:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:43:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:43:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:43:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:43:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:43:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:43:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:43:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:43:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:43:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:43:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:43:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:43:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:43:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:43:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:43:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2024-08-31T22:43:21.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:21.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_20 2024-08-31T22:43:21.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:21.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_20 2024-08-31T22:43:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_20 '' 2024-08-31T22:43:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_20 2024-08-31T22:43:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:43:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-08-31T22:43:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_20.status 2024-08-31T22:43:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_20.status 2024-08-31T22:43:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-08-31T22:43:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_20.status 2024-08-31T22:43:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:43:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:43:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8795], [object_number=10, tag_tid=1, entry_tid=8794]]], state=connected] 2024-08-31T22:43:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8795], [object_number=10, tag_tid=1, entry_tid=8794]]], state=connected] 2024-08-31T22:43:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_20.status 2024-08-31T22:43:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:43:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2024-08-31T22:43:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_20 2024-08-31T22:43:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2024-08-31T22:43:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_20 2024-08-31T22:43:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:43:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-08-31T22:43:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_20.status 2024-08-31T22:43:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:21.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_20.status 2024-08-31T22:43:21.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-08-31T22:43:21.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_20.status 2024-08-31T22:43:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:43:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:43:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8795], [object_number=10, tag_tid=1, entry_tid=8794]]], state=connected] 2024-08-31T22:43:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8795], [object_number=10, tag_tid=1, entry_tid=8794]]], state=connected] 2024-08-31T22:43:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_20.status 2024-08-31T22:43:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:43:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:43:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-08-31T22:43:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_20 2024-08-31T22:43:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:43:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-08-31T22:43:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_20': 2024-08-31T22:43:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:43:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:43:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:43:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10ab15bdecb1 2024-08-31T22:43:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10ab15bdecb1 2024-08-31T22:43:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:43:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:43:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:43:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:43:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:35:16 2024 2024-08-31T22:43:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:35:16 2024 2024-08-31T22:43:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:35:16 2024 2024-08-31T22:43:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:43:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10ab15bdecb1 2024-08-31T22:43:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:43:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:43:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 633373db-083a-4c29-8711-18786ed74c50 2024-08-31T22:43:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:43:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_20 snap 2024-08-31T22:43:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_20 2024-08-31T22:43:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:43:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_20.export 2024-08-31T22:43:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_20.export 2024-08-31T22:43:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_20.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_20.export 2024-08-31T22:43:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_20.export 2024-08-31T22:43:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_20.export 2024-08-31T22:43:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_20.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_20.export 2024-08-31T22:43:23.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_20.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_20.export 2024-08-31T22:43:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:43:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_21 2024-08-31T22:43:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_21 snap 2024-08-31T22:43:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_21 2024-08-31T22:43:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:43:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-08-31T22:43:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_21 2024-08-31T22:43:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-08-31T22:43:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_21 2024-08-31T22:43:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:43:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-08-31T22:43:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_21 2024-08-31T22:43:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:43:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:43:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:43:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-08-31T22:43:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:43:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:43:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:43:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:43:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:43:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:43:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-08-31T22:43:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:43:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:43:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:43:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:43:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:43:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:43:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:43:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-08-31T22:43:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:43:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:43:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:43:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:43:25.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:43:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:43:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:43:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-08-31T22:43:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_21 2024-08-31T22:43:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:43:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_21 2024-08-31T22:43:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_21 2024-08-31T22:43:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:43:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:43:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:43:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:43:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:43:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_21 2024-08-31T22:43:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_21 2024-08-31T22:43:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:43:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_21 ']' 2024-08-31T22:43:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_21' 2024-08-31T22:43:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2024-08-31T22:43:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:43:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:43:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:43:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:43:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:43:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2024-08-31T22:43:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:43:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:43:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:43:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:43:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:43:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:43:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:43:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:43:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:43:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:43:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:43:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:43:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:43:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2024-08-31T22:43:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_21 2024-08-31T22:43:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_21 2024-08-31T22:43:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_21 '' 2024-08-31T22:43:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_21 2024-08-31T22:43:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:43:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-08-31T22:43:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_21.status 2024-08-31T22:43:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_21.status 2024-08-31T22:43:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-08-31T22:43:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_21.status 2024-08-31T22:43:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:43:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:43:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4119], [object_number=6, tag_tid=1, entry_tid=4118]]], state=connected] 2024-08-31T22:43:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=4119], [object_number=6, tag_tid=1, entry_tid=4118], [object_number=5, tag_tid=1, entry_tid=4117], [object_number=4, tag_tid=1, entry_tid=4116]]], state=connected] 2024-08-31T22:43:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_21.status 2024-08-31T22:43:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:43:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2024-08-31T22:43:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_21 2024-08-31T22:43:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2024-08-31T22:43:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_21 2024-08-31T22:43:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:43:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-08-31T22:43:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_21.status 2024-08-31T22:43:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_21.status 2024-08-31T22:43:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-08-31T22:43:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_21.status 2024-08-31T22:43:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:43:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:43:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4119], [object_number=6, tag_tid=1, entry_tid=4118]]], state=connected] 2024-08-31T22:43:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=4119], [object_number=6, tag_tid=1, entry_tid=4118], [object_number=5, tag_tid=1, entry_tid=4117], [object_number=4, tag_tid=1, entry_tid=4116]]], state=connected] 2024-08-31T22:43:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_21.status 2024-08-31T22:43:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=7, tag_tid=1, entry_tid=4119' 2024-08-31T22:43:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=7, tag_tid=1, entry_tid=4119' 2024-08-31T22:43:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=4119' '!=' '' 2024-08-31T22:43:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:43:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=4119' = '' 2024-08-31T22:43:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=7, tag_tid=1, entry_tid=4119' 2024-08-31T22:43:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:43:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:43:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:43:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:43:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=7, tag_tid=1, entry_tid=4119' 2024-08-31T22:43:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:43:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:43:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=1 2024-08-31T22:43:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:43:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:43:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:43:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:43:25.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=7, tag_tid=1, entry_tid=4119' 2024-08-31T22:43:25.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:43:25.653 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:43:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=4119 2024-08-31T22:43:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 1 -a 1 -le 4119 2024-08-31T22:43:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:43:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:43:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:43:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_21 2024-08-31T22:43:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_21 2024-08-31T22:43:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:43:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_21 ']' 2024-08-31T22:43:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_21' 2024-08-31T22:43:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2024-08-31T22:43:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:43:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:43:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:43:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:43:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:43:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2024-08-31T22:43:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:43:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:43:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:43:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:43:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:43:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:43:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:43:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:43:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:43:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:43:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:43:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:43:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:43:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2024-08-31T22:43:26.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_21 2024-08-31T22:43:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_21 2024-08-31T22:43:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_21 '' 2024-08-31T22:43:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_21 2024-08-31T22:43:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:43:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-08-31T22:43:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_21.status 2024-08-31T22:43:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:26.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_21.status 2024-08-31T22:43:26.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-08-31T22:43:26.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_21.status 2024-08-31T22:43:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:43:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:43:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4119], [object_number=6, tag_tid=1, entry_tid=4118]]], state=connected] 2024-08-31T22:43:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4119], [object_number=6, tag_tid=1, entry_tid=4118]]], state=connected] 2024-08-31T22:43:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_21.status 2024-08-31T22:43:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:43:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2024-08-31T22:43:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_21 2024-08-31T22:43:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2024-08-31T22:43:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_21 2024-08-31T22:43:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:43:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-08-31T22:43:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_21.status 2024-08-31T22:43:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_21.status 2024-08-31T22:43:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-08-31T22:43:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_21.status 2024-08-31T22:43:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:43:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:43:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4119], [object_number=6, tag_tid=1, entry_tid=4118]]], state=connected] 2024-08-31T22:43:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4119], [object_number=6, tag_tid=1, entry_tid=4118]]], state=connected] 2024-08-31T22:43:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_21.status 2024-08-31T22:43:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:43:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:43:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-08-31T22:43:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_21 2024-08-31T22:43:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:43:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-08-31T22:43:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_21': 2024-08-31T22:43:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:43:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:43:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:43:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a83c79e210 2024-08-31T22:43:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a83c79e210 2024-08-31T22:43:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:43:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:43:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:43:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:43:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:35:27 2024 2024-08-31T22:43:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:35:27 2024 2024-08-31T22:43:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:36:28 2024 2024-08-31T22:43:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:43:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a83c79e210 2024-08-31T22:43:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:43:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:43:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: ec942725-b054-486c-a1ba-4d36facbedf0 2024-08-31T22:43:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:43:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_21 snap 2024-08-31T22:43:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_21 2024-08-31T22:43:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:43:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_21.export 2024-08-31T22:43:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_21.export 2024-08-31T22:43:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_21.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_21.export 2024-08-31T22:43:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_21.export 2024-08-31T22:43:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_21.export 2024-08-31T22:43:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_21.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_21.export 2024-08-31T22:43:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_21.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_21.export 2024-08-31T22:43:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:43:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_22 2024-08-31T22:43:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_22 snap 2024-08-31T22:43:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_22 2024-08-31T22:43:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:43:29.919 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:43:29.921+0000 7fdf7b115640 -1 reset not still connected to 0x55dde4411450 2024-08-31T22:43:29.921 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:43:29.921+0000 7fdf7b115640 -1 reset not still connected to 0x55dde49f60d0 2024-08-31T22:43:29.921 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:43:29.921+0000 7fdf7b115640 -1 reset not still connected to 0x55dde53556c0 2024-08-31T22:43:29.923 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:43:29.925+0000 7fdf7b115640 -1 reset not still connected to 0x55ddf45240d0 2024-08-31T22:43:29.923 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:43:29.925+0000 7fdf7b115640 -1 reset not still connected to 0x55ddfb2c9ad0 2024-08-31T22:43:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-08-31T22:43:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_22 2024-08-31T22:43:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-08-31T22:43:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_22 2024-08-31T22:43:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:43:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:30.734 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:43:30.737+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ec451e0 2024-08-31T22:43:30.746 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:43:30.737+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff16820 2024-08-31T22:43:30.747 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:43:30.741+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff16c30 2024-08-31T22:43:30.747 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:43:30.741+0000 7f13efcb9640 -1 reset not still connected to 0x559f1079bd40 2024-08-31T22:43:30.748 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:43:30.745+0000 7f13efcb9640 -1 reset not still connected to 0x559f13dc5040 2024-08-31T22:43:30.754 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:43:30.753+0000 7f13efcb9640 -1 reset not still connected to 0x559f14700b60 2024-08-31T22:43:30.754 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:43:30.753+0000 7f13efcb9640 -1 reset not still connected to 0x559f19235450 2024-08-31T22:43:30.754 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:43:30.757+0000 7f13efcb9640 -1 reset not still connected to 0x559f19a8ed00 2024-08-31T22:43:30.754 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:43:30.757+0000 7f13efcb9640 -1 reset not still connected to 0x559f21abe410 2024-08-31T22:43:30.754 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:43:30.757+0000 7f13efcb9640 -1 reset not still connected to 0x559f2246c750 2024-08-31T22:43:30.755 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:43:30.757+0000 7f13efcb9640 -1 reset not still connected to 0x559f24292410 2024-08-31T22:43:30.755 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:43:30.757+0000 7f13efcb9640 -1 reset not still connected to 0x559f273195f0 2024-08-31T22:43:30.755 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:43:30.757+0000 7f13efcb9640 -1 reset not still connected to 0x559f287c15f0 2024-08-31T22:43:30.755 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:43:30.757+0000 7f13efcb9640 -1 reset not still connected to 0x559f2b7404e0 2024-08-31T22:43:30.756 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:43:30.757+0000 7f13efcb9640 -1 reset not still connected to 0x559f2d6d9c70 2024-08-31T22:43:30.757 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:43:30.757+0000 7f13efcb9640 -1 reset not still connected to 0x559f363b1790 2024-08-31T22:43:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-08-31T22:43:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_22 2024-08-31T22:43:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:43:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:43:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:43:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-08-31T22:43:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:43:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:43:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:43:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:43:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:43:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:43:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-08-31T22:43:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:43:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:43:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:43:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:43:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:43:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:43:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:43:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:43:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:43:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:43:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:43:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:43:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:43:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-08-31T22:43:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-08-31T22:43:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:43:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:43:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-08-31T22:43:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:43:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:43:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:43:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:43:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:43:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:43:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:43:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:43:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:43:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:43:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:43:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:43:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:43:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:43:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:43:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:43:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:43:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:43:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:43:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:43:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:43:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:43:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-08-31T22:43:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:43:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:43:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:43:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:43:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:43:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:43:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:43:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-08-31T22:43:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_22 2024-08-31T22:43:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:43:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_22 2024-08-31T22:43:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_22 2024-08-31T22:43:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:43:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:43:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:43:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:43:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:43:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_22 2024-08-31T22:43:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_22 2024-08-31T22:43:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:43:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_22 ']' 2024-08-31T22:43:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_22' 2024-08-31T22:43:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_22 2024-08-31T22:43:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:43:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:43:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:43:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:43:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:43:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_22 2024-08-31T22:43:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:43:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:43:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:43:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:43:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:43:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:43:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:43:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:43:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:43:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:43:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:43:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:43:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:43:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_22 2024-08-31T22:43:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-08-31T22:43:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:43:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_22 2024-08-31T22:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:43:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:43:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:43:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:43:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:43:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:43:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:43:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:43:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:43:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:43:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:43:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:43:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:43:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:43:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:43:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_22 2024-08-31T22:43:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_22 2024-08-31T22:43:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_22 2024-08-31T22:43:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_22 '' 2024-08-31T22:43:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_22 2024-08-31T22:43:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:43:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-08-31T22:43:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_22.status 2024-08-31T22:43:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_22.status 2024-08-31T22:43:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-08-31T22:43:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_22.status 2024-08-31T22:43:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:43:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:43:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10667], [object_number=14, tag_tid=1, entry_tid=10666]]], state=connected] 2024-08-31T22:43:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10667], [object_number=14, tag_tid=1, entry_tid=10666]]], state=connected] 2024-08-31T22:43:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_22.status 2024-08-31T22:43:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:43:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_22 2024-08-31T22:43:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_22 2024-08-31T22:43:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_22 '..*' 2024-08-31T22:43:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_22 2024-08-31T22:43:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:43:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-08-31T22:43:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_22.status 2024-08-31T22:43:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_22.status 2024-08-31T22:43:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-08-31T22:43:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_22.status 2024-08-31T22:43:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:43:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:43:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10667], [object_number=14, tag_tid=1, entry_tid=10666]]], state=connected] 2024-08-31T22:43:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10667], [object_number=14, tag_tid=1, entry_tid=10666]]], state=connected] 2024-08-31T22:43:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_22.status 2024-08-31T22:43:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:43:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:43:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-08-31T22:43:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:31.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:31.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_22 2024-08-31T22:43:31.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:31.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:31.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:43:31.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-08-31T22:43:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_22': 2024-08-31T22:43:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:43:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:43:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:43:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10ab5701137d 2024-08-31T22:43:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10ab5701137d 2024-08-31T22:43:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:43:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:43:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:43:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:43:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:35:33 2024 2024-08-31T22:43:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:35:33 2024 2024-08-31T22:43:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:36:34 2024 2024-08-31T22:43:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:43:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10ab5701137d 2024-08-31T22:43:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:43:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:43:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 30656de8-9489-429b-ac9d-d8c736a473f1 2024-08-31T22:43:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:43:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_22 snap 2024-08-31T22:43:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_22 2024-08-31T22:43:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:43:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_22.export 2024-08-31T22:43:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_22.export 2024-08-31T22:43:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_22.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_22.export 2024-08-31T22:43:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_22.export 2024-08-31T22:43:33.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:33.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_22.export 2024-08-31T22:43:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:33.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_22.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_22.export 2024-08-31T22:43:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_22.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_22.export 2024-08-31T22:43:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:43:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_23 2024-08-31T22:43:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_23 snap 2024-08-31T22:43:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_23 2024-08-31T22:43:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:43:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-08-31T22:43:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_23 2024-08-31T22:43:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-08-31T22:43:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_23 2024-08-31T22:43:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:43:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-08-31T22:43:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_23 2024-08-31T22:43:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:43:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:43:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:43:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-08-31T22:43:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:43:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:43:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:43:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:43:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:43:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:43:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-08-31T22:43:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:43:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:43:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:43:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:43:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:43:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:43:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:43:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:43:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:43:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:43:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:43:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:43:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:43:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-08-31T22:43:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:43:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:43:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:43:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:43:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:43:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:43:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:43:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-08-31T22:43:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_23 2024-08-31T22:43:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:43:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_23 2024-08-31T22:43:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_23 2024-08-31T22:43:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:43:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:43:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:43:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:43:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:43:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_23 2024-08-31T22:43:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_23 2024-08-31T22:43:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:43:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_23 ']' 2024-08-31T22:43:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_23' 2024-08-31T22:43:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_23 2024-08-31T22:43:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:43:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:43:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:43:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:43:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:43:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_23 2024-08-31T22:43:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:43:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:43:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:43:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:43:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:43:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:43:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:43:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:43:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:43:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:43:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:43:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:43:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:43:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:37.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:37.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:37.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_23 2024-08-31T22:43:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_23 2024-08-31T22:43:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_23 2024-08-31T22:43:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_23 '' 2024-08-31T22:43:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_23 2024-08-31T22:43:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:43:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-08-31T22:43:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_23.status 2024-08-31T22:43:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_23.status 2024-08-31T22:43:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-08-31T22:43:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_23.status 2024-08-31T22:43:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:43:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:43:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10427], [object_number=10, tag_tid=1, entry_tid=10426]]], state=connected] 2024-08-31T22:43:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10427], [object_number=10, tag_tid=1, entry_tid=10426]]], state=connected] 2024-08-31T22:43:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_23.status 2024-08-31T22:43:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:43:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_23 2024-08-31T22:43:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_23 2024-08-31T22:43:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_23 '..*' 2024-08-31T22:43:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_23 2024-08-31T22:43:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:43:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-08-31T22:43:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_23.status 2024-08-31T22:43:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_23.status 2024-08-31T22:43:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-08-31T22:43:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_23.status 2024-08-31T22:43:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:43:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:43:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10427], [object_number=10, tag_tid=1, entry_tid=10426]]], state=connected] 2024-08-31T22:43:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10427], [object_number=10, tag_tid=1, entry_tid=10426]]], state=connected] 2024-08-31T22:43:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_23.status 2024-08-31T22:43:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:43:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:43:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-08-31T22:43:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_23 2024-08-31T22:43:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:43:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-08-31T22:43:38.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_23': 2024-08-31T22:43:38.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:43:38.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:43:38.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:43:38.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a88fbb3344 2024-08-31T22:43:38.954 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a88fbb3344 2024-08-31T22:43:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:43:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:43:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:43:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:43:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:35:46 2024 2024-08-31T22:43:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:35:46 2024 2024-08-31T22:43:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:36:47 2024 2024-08-31T22:43:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:43:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a88fbb3344 2024-08-31T22:43:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:43:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:43:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 8dd32205-1c45-4d2b-ad0f-8767405c3074 2024-08-31T22:43:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:43:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:38.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_23 snap 2024-08-31T22:43:38.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:38.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_23 2024-08-31T22:43:38.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:38.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:43:38.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_23.export 2024-08-31T22:43:38.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_23.export 2024-08-31T22:43:38.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_23.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_23.export 2024-08-31T22:43:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_23.export 2024-08-31T22:43:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_23.export 2024-08-31T22:43:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_23.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_23.export 2024-08-31T22:43:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_23.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_23.export 2024-08-31T22:43:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:43:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_24 2024-08-31T22:43:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_24 snap 2024-08-31T22:43:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_24 2024-08-31T22:43:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:43:41.485 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:43:41.484+0000 7ff273ffe640 -1 reset not still connected to 0x5619515f5110 2024-08-31T22:43:41.486 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:43:41.484+0000 7ff273ffe640 -1 reset not still connected to 0x561954ef31e0 2024-08-31T22:43:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-08-31T22:43:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_24 2024-08-31T22:43:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-08-31T22:43:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_24 2024-08-31T22:43:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:43:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-08-31T22:43:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_24 2024-08-31T22:43:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:43:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:43:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:43:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-08-31T22:43:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:43:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:43:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:43:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:43:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:43:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:43:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-08-31T22:43:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:43:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:43:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:43:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:43:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:43:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:43:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:43:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:43:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:43:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:43:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:43:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:43:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:43:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-08-31T22:43:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-08-31T22:43:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:43:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:43:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2024-08-31T22:43:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:43:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:43:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:43:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:43:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:43:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:43:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:43:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:43:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:43:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:43:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:43:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:43:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:43:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:43:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:43:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:43:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:43:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:43:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:43:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:43:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:43:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:43:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2024-08-31T22:43:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:43:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:43:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:43:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:43:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:43:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:43:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:43:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-08-31T22:43:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_24 2024-08-31T22:43:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:43:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_24 2024-08-31T22:43:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_24 2024-08-31T22:43:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:43:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:43:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:43:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:43:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:43:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_24 2024-08-31T22:43:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_24 2024-08-31T22:43:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:43:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_24 ']' 2024-08-31T22:43:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_24' 2024-08-31T22:43:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_24 2024-08-31T22:43:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:43:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:43:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:43:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:43:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:43:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_24 2024-08-31T22:43:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:43:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:43:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:43:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:43:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:43:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:43:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:43:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:43:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:43:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:43:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:43:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:43:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:43:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_24 2024-08-31T22:43:43.991 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:43:43.992+0000 7fdf7b115640 -1 reset not still connected to 0x55dde4411450 2024-08-31T22:43:43.997 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:43:43.996+0000 7fdf7b115640 -1 reset not still connected to 0x55dde49f60d0 2024-08-31T22:43:43.997 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:43:43.996+0000 7fdf7b115640 -1 reset not still connected to 0x55ddf45240d0 2024-08-31T22:43:43.997 INFO:tasks.ceph.cluster2.osd.1.smithi032.stderr:2024-08-31T22:43:43.996+0000 7fdf7b115640 -1 reset not still connected to 0x55ddfb2c9ad0 2024-08-31T22:43:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-08-31T22:43:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:43:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:43:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_24 2024-08-31T22:43:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:43:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:43:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:43:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:43:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:43:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:43:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:43:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:43:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:43:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:43:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:43:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:43:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:43:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:43:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:43:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:43:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:43:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:43:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:43:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:43:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:43:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:43:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_24 2024-08-31T22:43:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_24 2024-08-31T22:43:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_24 2024-08-31T22:43:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_24 '' 2024-08-31T22:43:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_24 2024-08-31T22:43:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:43:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-08-31T22:43:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_24.status 2024-08-31T22:43:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_24.status 2024-08-31T22:43:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-08-31T22:43:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_24.status 2024-08-31T22:43:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:43:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:43:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11023], [object_number=14, tag_tid=1, entry_tid=11022]]], state=connected] 2024-08-31T22:43:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11023], [object_number=14, tag_tid=1, entry_tid=11022]]], state=connected] 2024-08-31T22:43:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_24.status 2024-08-31T22:43:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:43:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_24 2024-08-31T22:43:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_24 2024-08-31T22:43:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_24 '..*' 2024-08-31T22:43:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_24 2024-08-31T22:43:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:43:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-08-31T22:43:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_24.status 2024-08-31T22:43:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_24.status 2024-08-31T22:43:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-08-31T22:43:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_24.status 2024-08-31T22:43:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:43:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:43:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11023], [object_number=14, tag_tid=1, entry_tid=11022]]], state=connected] 2024-08-31T22:43:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11023], [object_number=14, tag_tid=1, entry_tid=11022]]], state=connected] 2024-08-31T22:43:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_24.status 2024-08-31T22:43:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:43:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:43:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-08-31T22:43:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_24 2024-08-31T22:43:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:43:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-08-31T22:43:45.350 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:43:45.348+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff161a0 2024-08-31T22:43:45.350 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:43:45.348+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff16d00 2024-08-31T22:43:45.350 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:43:45.348+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff17c70 2024-08-31T22:43:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_24': 2024-08-31T22:43:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:43:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:43:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:43:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10ab3255797b 2024-08-31T22:43:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10ab3255797b 2024-08-31T22:43:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:43:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:43:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:43:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:43:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:35:57 2024 2024-08-31T22:43:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:35:57 2024 2024-08-31T22:43:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:36:58 2024 2024-08-31T22:43:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:43:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10ab3255797b 2024-08-31T22:43:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:43:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:43:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 4f949067-b588-412a-82a4-97d250e2a9d4 2024-08-31T22:43:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:43:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_24 snap 2024-08-31T22:43:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_24 2024-08-31T22:43:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:43:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_24.export 2024-08-31T22:43:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_24.export 2024-08-31T22:43:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_24.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_24.export 2024-08-31T22:43:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_24.export 2024-08-31T22:43:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_24.export 2024-08-31T22:43:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:45.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_24.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_24.export 2024-08-31T22:43:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_24.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_24.export 2024-08-31T22:43:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:43:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_25 2024-08-31T22:43:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_25 snap 2024-08-31T22:43:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_25 2024-08-31T22:43:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:43:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-08-31T22:43:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_25 2024-08-31T22:43:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-08-31T22:43:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_25 2024-08-31T22:43:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:43:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:47.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:47.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-08-31T22:43:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_25 2024-08-31T22:43:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:43:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:43:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:43:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-08-31T22:43:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:43:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:43:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:43:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:43:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:43:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:43:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-08-31T22:43:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:43:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:43:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:43:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:43:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:43:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:43:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:43:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:43:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:43:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:43:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:43:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:43:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:43:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-08-31T22:43:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:43:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:43:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:43:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:43:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:43:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:43:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:43:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-08-31T22:43:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_25 2024-08-31T22:43:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:43:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_25 2024-08-31T22:43:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_25 2024-08-31T22:43:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:43:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:43:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:43:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:43:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:43:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_25 2024-08-31T22:43:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_25 2024-08-31T22:43:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:43:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_25 ']' 2024-08-31T22:43:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_25' 2024-08-31T22:43:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2024-08-31T22:43:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:43:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:43:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:43:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:43:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:43:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2024-08-31T22:43:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:43:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:43:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:43:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:43:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:43:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:43:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:43:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:43:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:43:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:43:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:43:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:43:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:43:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2024-08-31T22:43:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_25 2024-08-31T22:43:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_25 2024-08-31T22:43:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_25 '' 2024-08-31T22:43:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_25 2024-08-31T22:43:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:43:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-08-31T22:43:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_25.status 2024-08-31T22:43:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_25.status 2024-08-31T22:43:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-08-31T22:43:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_25.status 2024-08-31T22:43:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 0 2024-08-31T22:43:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:43:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3354], [object_number=3, tag_tid=1, entry_tid=3351]]], state=connected] 2024-08-31T22:43:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3354], [object_number=3, tag_tid=1, entry_tid=3351]]], state=connected] 2024-08-31T22:43:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_25.status 2024-08-31T22:43:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:43:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2024-08-31T22:43:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_25 2024-08-31T22:43:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2024-08-31T22:43:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_25 2024-08-31T22:43:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:43:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-08-31T22:43:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_25.status 2024-08-31T22:43:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_25.status 2024-08-31T22:43:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-08-31T22:43:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_25.status 2024-08-31T22:43:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 0 2024-08-31T22:43:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:43:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3354], [object_number=3, tag_tid=1, entry_tid=3351]]], state=connected] 2024-08-31T22:43:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3354], [object_number=3, tag_tid=1, entry_tid=3351]]], state=connected] 2024-08-31T22:43:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_25.status 2024-08-31T22:43:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:43:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:43:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-08-31T22:43:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_25 2024-08-31T22:43:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:43:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:50.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-08-31T22:43:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_25': 2024-08-31T22:43:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:43:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:43:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:43:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a8fa6b55c8 2024-08-31T22:43:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a8fa6b55c8 2024-08-31T22:43:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:43:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:43:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:43:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:43:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:36:09 2024 2024-08-31T22:43:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:36:09 2024 2024-08-31T22:43:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:36:09 2024 2024-08-31T22:43:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:43:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a8fa6b55c8 2024-08-31T22:43:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:43:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:43:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 7d68c308-db49-44e3-b75e-f04698a6f032 2024-08-31T22:43:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:43:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_25 snap 2024-08-31T22:43:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_25 2024-08-31T22:43:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:43:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_25.export 2024-08-31T22:43:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_25.export 2024-08-31T22:43:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_25.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_25.export 2024-08-31T22:43:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_25.export 2024-08-31T22:43:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_25.export 2024-08-31T22:43:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_25.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_25.export 2024-08-31T22:43:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_25.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_25.export 2024-08-31T22:43:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:43:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_26 2024-08-31T22:43:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_26 snap 2024-08-31T22:43:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_26 2024-08-31T22:43:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:43:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-08-31T22:43:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_26 2024-08-31T22:43:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-08-31T22:43:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_26 2024-08-31T22:43:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:43:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:53.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-08-31T22:43:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_26 2024-08-31T22:43:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:43:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:43:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:43:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-08-31T22:43:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:43:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:43:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:43:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:43:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:43:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:43:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-08-31T22:43:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:43:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:43:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:43:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:43:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:43:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:43:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:43:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:43:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:43:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:43:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:43:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:43:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:43:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-08-31T22:43:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-08-31T22:43:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:43:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:43:53.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-08-31T22:43:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:43:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:43:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:43:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:43:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:43:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:43:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:43:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:43:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:43:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:43:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:43:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:43:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:43:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:43:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:43:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:43:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:43:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:43:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:43:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:43:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:43:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:43:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-08-31T22:43:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:43:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:43:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:43:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:43:53.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:43:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:43:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:43:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-08-31T22:43:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_26 2024-08-31T22:43:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:43:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_26 2024-08-31T22:43:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_26 2024-08-31T22:43:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:43:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:43:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:43:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:43:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:43:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_26 2024-08-31T22:43:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_26 2024-08-31T22:43:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:43:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_26 ']' 2024-08-31T22:43:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_26' 2024-08-31T22:43:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_26 2024-08-31T22:43:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:43:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:43:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:43:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:43:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:43:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_26 2024-08-31T22:43:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:43:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:43:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:43:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:43:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:43:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:43:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:43:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:43:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:43:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:43:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:43:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:43:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:43:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_26 2024-08-31T22:43:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-08-31T22:43:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:43:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:43:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_26 2024-08-31T22:43:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:43:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:43:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:43:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:43:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:43:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:43:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:43:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:43:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:43:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:43:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:43:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:43:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:43:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:43:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:43:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:43:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:43:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:43:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:43:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:43:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:43:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:43:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:43:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:43:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_26 2024-08-31T22:43:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_26 2024-08-31T22:43:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_26 2024-08-31T22:43:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_26 '' 2024-08-31T22:43:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_26 2024-08-31T22:43:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:43:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_26.status 2024-08-31T22:43:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_26.status 2024-08-31T22:43:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_26.status 2024-08-31T22:43:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_26.status 2024-08-31T22:43:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2024-08-31T22:43:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:43:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:43:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5483], [object_number=6, tag_tid=1, entry_tid=5482]]], state=connected] 2024-08-31T22:43:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5483], [object_number=6, tag_tid=1, entry_tid=5482]]], state=connected] 2024-08-31T22:43:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_26.status 2024-08-31T22:43:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:43:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_26 2024-08-31T22:43:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_26 2024-08-31T22:43:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_26 '..*' 2024-08-31T22:43:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_26 2024-08-31T22:43:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:43:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_26.status 2024-08-31T22:43:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_26.status 2024-08-31T22:43:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_26.status 2024-08-31T22:43:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2024-08-31T22:43:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_26.status 2024-08-31T22:43:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:43:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:43:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5483], [object_number=6, tag_tid=1, entry_tid=5482]]], state=connected] 2024-08-31T22:43:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5483], [object_number=6, tag_tid=1, entry_tid=5482]]], state=connected] 2024-08-31T22:43:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_26.status 2024-08-31T22:43:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:43:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:43:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-08-31T22:43:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_26 2024-08-31T22:43:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:43:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-08-31T22:43:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_26': 2024-08-31T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10ab71816c43 2024-08-31T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10ab71816c43 2024-08-31T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:36:15 2024 2024-08-31T22:43:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:36:15 2024 2024-08-31T22:43:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:36:15 2024 2024-08-31T22:43:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:43:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10ab71816c43 2024-08-31T22:43:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:43:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:43:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 4c9c31f6-7d2c-481c-8255-90cddb178837 2024-08-31T22:43:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:43:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_26 snap 2024-08-31T22:43:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_26 2024-08-31T22:43:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:43:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_26.export 2024-08-31T22:43:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_26.export 2024-08-31T22:43:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_26.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_26.export 2024-08-31T22:43:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_26.export 2024-08-31T22:43:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_26.export 2024-08-31T22:43:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:43:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_26.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_26.export 2024-08-31T22:43:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_26.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_26.export 2024-08-31T22:43:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:43:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_27 2024-08-31T22:43:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_27 snap 2024-08-31T22:43:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_27 2024-08-31T22:43:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:43:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-08-31T22:43:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_27 2024-08-31T22:43:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-08-31T22:43:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_27 2024-08-31T22:43:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:43:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:43:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-08-31T22:43:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_27 2024-08-31T22:43:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:43:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:43:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:43:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-08-31T22:43:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:43:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:43:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:43:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:43:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:43:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:43:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-08-31T22:43:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:43:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:43:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:43:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:43:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:43:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:43:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:43:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:43:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:43:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:43:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:43:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:43:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:43:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:43:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:43:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:58.371 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-08-31T22:43:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:43:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:43:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:43:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:43:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:43:58.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:43:58.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:43:58.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:58.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-08-31T22:43:58.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:58.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:58.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:58.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_27 2024-08-31T22:43:58.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:43:58.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_27 2024-08-31T22:43:58.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:43:58.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:43:58.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:58.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_27 2024-08-31T22:43:58.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:43:58.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:43:58.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:43:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:43:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_27 2024-08-31T22:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_27 2024-08-31T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_27 ']' 2024-08-31T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_27' 2024-08-31T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2024-08-31T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:43:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:43:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:43:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:43:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2024-08-31T22:43:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:43:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:43:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:43:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:43:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:43:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:43:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:43:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:43:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:43:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:43:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:43:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:43:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:43:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:43:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:43:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:43:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:43:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:43:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:43:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:43:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:43:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:43:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:43:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:43:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:43:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:43:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:43:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:43:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2024-08-31T22:43:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_27 2024-08-31T22:43:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_27 2024-08-31T22:43:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_27 '' 2024-08-31T22:43:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_27 2024-08-31T22:43:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:43:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-08-31T22:43:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_27.status 2024-08-31T22:43:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_27.status 2024-08-31T22:43:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_27.status 2024-08-31T22:43:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-08-31T22:43:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:43:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:43:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=3598], [object_number=7, tag_tid=1, entry_tid=3595]]], state=connected] 2024-08-31T22:43:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=3598], [object_number=7, tag_tid=1, entry_tid=3595]]], state=connected] 2024-08-31T22:43:58.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_27.status 2024-08-31T22:43:58.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:43:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2024-08-31T22:43:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_27 2024-08-31T22:43:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2024-08-31T22:43:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:43:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:43:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_27 2024-08-31T22:43:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:43:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-08-31T22:43:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_27.status 2024-08-31T22:43:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:43:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_27.status 2024-08-31T22:43:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-08-31T22:43:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_27.status 2024-08-31T22:43:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:43:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:43:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:43:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=3598], [object_number=7, tag_tid=1, entry_tid=3595]]], state=connected] 2024-08-31T22:43:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=3598], [object_number=7, tag_tid=1, entry_tid=3595]]], state=connected] 2024-08-31T22:43:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_27.status 2024-08-31T22:43:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:43:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:43:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:43:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-08-31T22:43:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:43:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:43:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_27 2024-08-31T22:43:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:43:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:43:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:43:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-08-31T22:44:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_27': 2024-08-31T22:44:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:44:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:44:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:44:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a877d46cc6 2024-08-31T22:44:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a877d46cc6 2024-08-31T22:44:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:44:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:44:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:44:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:44:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:36:23 2024 2024-08-31T22:44:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:36:23 2024 2024-08-31T22:44:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:37:23 2024 2024-08-31T22:44:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:44:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a877d46cc6 2024-08-31T22:44:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:44:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:44:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: a296ef38-5d0a-4942-a129-1b051155e824 2024-08-31T22:44:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:44:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_27 snap 2024-08-31T22:44:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_27 2024-08-31T22:44:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:44:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_27.export 2024-08-31T22:44:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_27.export 2024-08-31T22:44:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_27.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_27.export 2024-08-31T22:44:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_27.export 2024-08-31T22:44:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_27.export 2024-08-31T22:44:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_27.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_27.export 2024-08-31T22:44:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_27.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_27.export 2024-08-31T22:44:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:44:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_28 2024-08-31T22:44:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_28 snap 2024-08-31T22:44:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_28 2024-08-31T22:44:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:44:03.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-08-31T22:44:03.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:03.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_28 2024-08-31T22:44:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-08-31T22:44:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_28 2024-08-31T22:44:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:44:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-08-31T22:44:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_28 2024-08-31T22:44:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:44:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:44:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:44:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-08-31T22:44:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:44:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:44:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:44:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:44:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:44:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:44:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-08-31T22:44:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:44:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:44:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:44:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:44:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:44:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:44:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:44:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:44:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:44:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:44:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:44:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:44:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:44:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-08-31T22:44:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-08-31T22:44:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:44:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:44:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2024-08-31T22:44:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:44:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:44:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:44:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:44:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:44:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:44:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:44:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:44:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:44:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:44:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:44:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:44:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:44:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:44:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:44:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:44:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:44:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:44:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:44:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:44:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:44:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:44:04.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2024-08-31T22:44:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:44:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:44:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:44:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:44:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:44:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:44:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:44:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-08-31T22:44:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_28 2024-08-31T22:44:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:44:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_28 2024-08-31T22:44:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_28 2024-08-31T22:44:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:44:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:44:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:44:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:44:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:44:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_28 2024-08-31T22:44:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_28 2024-08-31T22:44:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:44:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_28 ']' 2024-08-31T22:44:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_28' 2024-08-31T22:44:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_28 2024-08-31T22:44:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:44:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:44:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:44:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:44:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:44:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_28 2024-08-31T22:44:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:44:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:44:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:44:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:44:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:44:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:44:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:44:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:44:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:44:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:44:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:44:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:44:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:44:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_28 2024-08-31T22:44:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-08-31T22:44:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:44:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:44:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_28 2024-08-31T22:44:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:44:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:44:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:44:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:44:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:44:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:44:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:44:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:44:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:44:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:44:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:44:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:44:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:44:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:44:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:44:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:44:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:44:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:44:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:44:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:44:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:44:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:44:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_28 2024-08-31T22:44:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:05.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_28 2024-08-31T22:44:05.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:05.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:05.197 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_28 2024-08-31T22:44:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_28 '' 2024-08-31T22:44:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_28 2024-08-31T22:44:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:44:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-08-31T22:44:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_28.status 2024-08-31T22:44:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_28.status 2024-08-31T22:44:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-08-31T22:44:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_28.status 2024-08-31T22:44:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:44:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:44:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9635], [object_number=10, tag_tid=1, entry_tid=9634]]], state=connected] 2024-08-31T22:44:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9635], [object_number=10, tag_tid=1, entry_tid=9634]]], state=connected] 2024-08-31T22:44:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_28.status 2024-08-31T22:44:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:44:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_28 2024-08-31T22:44:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_28 2024-08-31T22:44:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_28 '..*' 2024-08-31T22:44:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_28 2024-08-31T22:44:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:44:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-08-31T22:44:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_28.status 2024-08-31T22:44:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_28.status 2024-08-31T22:44:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-08-31T22:44:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_28.status 2024-08-31T22:44:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:44:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:44:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9635], [object_number=10, tag_tid=1, entry_tid=9634]]], state=connected] 2024-08-31T22:44:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9635], [object_number=10, tag_tid=1, entry_tid=9634]]], state=connected] 2024-08-31T22:44:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_28.status 2024-08-31T22:44:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:44:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:44:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-08-31T22:44:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_28 2024-08-31T22:44:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:44:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-08-31T22:44:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_28': 2024-08-31T22:44:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:44:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:44:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:44:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10abfe102a43 2024-08-31T22:44:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10abfe102a43 2024-08-31T22:44:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:44:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:44:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:44:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:44:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:36:28 2024 2024-08-31T22:44:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:36:28 2024 2024-08-31T22:44:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:37:29 2024 2024-08-31T22:44:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:44:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10abfe102a43 2024-08-31T22:44:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:44:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:44:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 8afdc41f-3fcc-47c6-b091-5ac5c1672c70 2024-08-31T22:44:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:44:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:06.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_28 snap 2024-08-31T22:44:06.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:06.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_28 2024-08-31T22:44:06.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:06.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:44:06.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_28.export 2024-08-31T22:44:06.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_28.export 2024-08-31T22:44:06.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_28.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_28.export 2024-08-31T22:44:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_28.export 2024-08-31T22:44:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_28.export 2024-08-31T22:44:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_28.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_28.export 2024-08-31T22:44:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_28.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_28.export 2024-08-31T22:44:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:44:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_29 2024-08-31T22:44:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_29 snap 2024-08-31T22:44:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_29 2024-08-31T22:44:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:44:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-08-31T22:44:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_29 2024-08-31T22:44:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-08-31T22:44:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_29 2024-08-31T22:44:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:44:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-08-31T22:44:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_29 2024-08-31T22:44:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:44:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:44:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:44:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-08-31T22:44:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:44:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:44:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:44:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:44:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:44:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:44:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-08-31T22:44:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:44:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:44:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:44:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:44:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:44:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:44:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:44:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:44:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:44:10.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:44:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:44:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:44:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:44:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:10.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:10.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:10.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:10.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-08-31T22:44:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:44:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:44:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:44:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:44:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:44:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:44:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:44:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-08-31T22:44:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_29 2024-08-31T22:44:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:44:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_29 2024-08-31T22:44:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_29 2024-08-31T22:44:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:44:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:44:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:44:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:44:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:44:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_29 2024-08-31T22:44:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_29 2024-08-31T22:44:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:44:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_29 ']' 2024-08-31T22:44:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_29' 2024-08-31T22:44:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_29 2024-08-31T22:44:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:44:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:44:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:44:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:44:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:44:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_29 2024-08-31T22:44:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:44:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:44:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:44:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:44:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:44:10.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:44:10.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:44:10.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:10.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:44:10.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:10.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:44:10.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:44:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:44:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:44:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:44:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_29 2024-08-31T22:44:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_29 2024-08-31T22:44:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_29 2024-08-31T22:44:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_29 '' 2024-08-31T22:44:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_29 2024-08-31T22:44:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:44:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-08-31T22:44:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_29.status 2024-08-31T22:44:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_29.status 2024-08-31T22:44:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-08-31T22:44:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_29.status 2024-08-31T22:44:10.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:44:10.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:44:10.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:10.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10639], [object_number=14, tag_tid=1, entry_tid=10638]]], state=connected] 2024-08-31T22:44:10.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10639], [object_number=14, tag_tid=1, entry_tid=10638]]], state=connected] 2024-08-31T22:44:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_29.status 2024-08-31T22:44:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_29 2024-08-31T22:44:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_29 2024-08-31T22:44:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_29 '..*' 2024-08-31T22:44:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_29 2024-08-31T22:44:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:44:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-08-31T22:44:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_29.status 2024-08-31T22:44:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_29.status 2024-08-31T22:44:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-08-31T22:44:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_29.status 2024-08-31T22:44:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:44:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:44:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10639], [object_number=14, tag_tid=1, entry_tid=10638]]], state=connected] 2024-08-31T22:44:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10639], [object_number=14, tag_tid=1, entry_tid=10638]]], state=connected] 2024-08-31T22:44:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_29.status 2024-08-31T22:44:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-08-31T22:44:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_29 2024-08-31T22:44:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:44:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:12.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-08-31T22:44:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_29': 2024-08-31T22:44:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:44:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:44:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:44:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a82f1ff5b9 2024-08-31T22:44:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a82f1ff5b9 2024-08-31T22:44:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:44:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:44:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:44:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:44:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:36:38 2024 2024-08-31T22:44:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:36:38 2024 2024-08-31T22:44:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:37:39 2024 2024-08-31T22:44:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:44:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a82f1ff5b9 2024-08-31T22:44:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:44:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:44:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 25532b02-453c-4714-b4cd-0b3043467bc2 2024-08-31T22:44:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:44:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_29 snap 2024-08-31T22:44:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_29 2024-08-31T22:44:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:44:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_29.export 2024-08-31T22:44:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_29.export 2024-08-31T22:44:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_29.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_29.export 2024-08-31T22:44:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_29.export 2024-08-31T22:44:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_29.export 2024-08-31T22:44:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_29.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_29.export 2024-08-31T22:44:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_29.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_29.export 2024-08-31T22:44:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:44:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_30 2024-08-31T22:44:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_30 snap 2024-08-31T22:44:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_30 2024-08-31T22:44:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:44:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-08-31T22:44:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_30 2024-08-31T22:44:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-08-31T22:44:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_30 2024-08-31T22:44:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:44:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-08-31T22:44:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_30 2024-08-31T22:44:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:44:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:44:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:44:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-08-31T22:44:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:44:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:44:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:44:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:44:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:44:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:44:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-08-31T22:44:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:44:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:44:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:44:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:44:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:44:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:44:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:44:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:44:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:44:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:44:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:44:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:44:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:44:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-08-31T22:44:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-08-31T22:44:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:44:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:44:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2024-08-31T22:44:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:44:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:44:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:44:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:44:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:44:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:44:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:44:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:44:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:44:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:44:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:44:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:44:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:44:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:44:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:44:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:44:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:44:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:44:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:44:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:44:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:44:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:44:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2024-08-31T22:44:16.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:44:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:44:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:44:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:44:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:44:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:44:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:44:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-08-31T22:44:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_30 2024-08-31T22:44:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:44:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_30 2024-08-31T22:44:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_30 2024-08-31T22:44:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:44:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:44:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:44:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:44:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:44:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_30 2024-08-31T22:44:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_30 2024-08-31T22:44:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:44:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_30 ']' 2024-08-31T22:44:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_30' 2024-08-31T22:44:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_30 2024-08-31T22:44:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:44:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:44:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:44:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:44:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:44:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_30 2024-08-31T22:44:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:44:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:44:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:44:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:44:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:44:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:44:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:44:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:44:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:44:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:44:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:44:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:44:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:44:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_30 2024-08-31T22:44:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-08-31T22:44:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:44:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:44:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_30 2024-08-31T22:44:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:44:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:44:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:44:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:44:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:44:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:44:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:44:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:44:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:44:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:44:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:44:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:44:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:44:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:44:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:44:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:44:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:44:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:44:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:44:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:44:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:44:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:44:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_30 2024-08-31T22:44:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_30 2024-08-31T22:44:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_30 2024-08-31T22:44:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_30 '' 2024-08-31T22:44:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_30 2024-08-31T22:44:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:44:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-08-31T22:44:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_30.status 2024-08-31T22:44:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_30.status 2024-08-31T22:44:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-08-31T22:44:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_30.status 2024-08-31T22:44:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:44:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:44:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13590], [object_number=15, tag_tid=1, entry_tid=13587]]], state=connected] 2024-08-31T22:44:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13590], [object_number=15, tag_tid=1, entry_tid=13587]]], state=connected] 2024-08-31T22:44:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_30.status 2024-08-31T22:44:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_30 2024-08-31T22:44:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_30 2024-08-31T22:44:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_30 '..*' 2024-08-31T22:44:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_30 2024-08-31T22:44:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:44:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-08-31T22:44:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_30.status 2024-08-31T22:44:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_30.status 2024-08-31T22:44:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-08-31T22:44:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_30.status 2024-08-31T22:44:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:44:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:44:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13590], [object_number=15, tag_tid=1, entry_tid=13587]]], state=connected] 2024-08-31T22:44:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13590], [object_number=15, tag_tid=1, entry_tid=13587]]], state=connected] 2024-08-31T22:44:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_30.status 2024-08-31T22:44:17.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:17.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:17.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-08-31T22:44:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_30 2024-08-31T22:44:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:44:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-08-31T22:44:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_30': 2024-08-31T22:44:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:44:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:44:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:44:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10ab804383fe 2024-08-31T22:44:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10ab804383fe 2024-08-31T22:44:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:44:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:44:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:44:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:44:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:36:51 2024 2024-08-31T22:44:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:36:51 2024 2024-08-31T22:44:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:37:52 2024 2024-08-31T22:44:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:44:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10ab804383fe 2024-08-31T22:44:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:44:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:44:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 7120f134-0a5c-4301-8901-2f5340fb9063 2024-08-31T22:44:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:44:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_30 snap 2024-08-31T22:44:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_30 2024-08-31T22:44:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:44:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_30.export 2024-08-31T22:44:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_30.export 2024-08-31T22:44:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_30.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_30.export 2024-08-31T22:44:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_30.export 2024-08-31T22:44:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_30.export 2024-08-31T22:44:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_30.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_30.export 2024-08-31T22:44:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_30.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_30.export 2024-08-31T22:44:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:44:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_31 2024-08-31T22:44:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_31 snap 2024-08-31T22:44:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_31 2024-08-31T22:44:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:44:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-08-31T22:44:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_31 2024-08-31T22:44:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-08-31T22:44:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_31 2024-08-31T22:44:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:44:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-08-31T22:44:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_31 2024-08-31T22:44:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:44:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:44:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:44:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-08-31T22:44:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:44:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:44:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:44:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:44:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:44:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:44:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-08-31T22:44:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:44:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:44:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:44:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:44:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:44:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:44:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:44:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:44:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:44:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:44:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:44:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:44:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:44:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-08-31T22:44:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:44:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:44:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:44:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:44:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:44:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:44:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:44:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-08-31T22:44:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_31 2024-08-31T22:44:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:44:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_31 2024-08-31T22:44:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_31 2024-08-31T22:44:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:44:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:44:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:44:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:44:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:44:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_31 2024-08-31T22:44:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_31 2024-08-31T22:44:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:44:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_31 ']' 2024-08-31T22:44:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_31' 2024-08-31T22:44:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2024-08-31T22:44:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:44:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:44:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:44:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:44:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:44:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2024-08-31T22:44:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:44:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:44:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:44:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:44:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:44:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:44:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:44:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:44:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:44:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:44:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:44:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:44:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:44:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2024-08-31T22:44:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_31 2024-08-31T22:44:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_31 2024-08-31T22:44:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_31 '' 2024-08-31T22:44:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_31 2024-08-31T22:44:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:44:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-08-31T22:44:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_31.status 2024-08-31T22:44:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:22.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_31.status 2024-08-31T22:44:22.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-08-31T22:44:22.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_31.status 2024-08-31T22:44:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:44:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:44:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9875], [object_number=10, tag_tid=1, entry_tid=9874]]], state=connected] 2024-08-31T22:44:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9875], [object_number=10, tag_tid=1, entry_tid=9874]]], state=connected] 2024-08-31T22:44:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_31.status 2024-08-31T22:44:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:44:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2024-08-31T22:44:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_31 2024-08-31T22:44:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2024-08-31T22:44:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_31 2024-08-31T22:44:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:44:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-08-31T22:44:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_31.status 2024-08-31T22:44:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_31.status 2024-08-31T22:44:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-08-31T22:44:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_31.status 2024-08-31T22:44:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:44:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:44:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9875], [object_number=10, tag_tid=1, entry_tid=9874]]], state=connected] 2024-08-31T22:44:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9875], [object_number=10, tag_tid=1, entry_tid=9874]]], state=connected] 2024-08-31T22:44:23.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_31.status 2024-08-31T22:44:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:44:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:44:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-08-31T22:44:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_31 2024-08-31T22:44:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:44:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-08-31T22:44:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_31': 2024-08-31T22:44:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:44:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:44:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:44:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a8cb31317a 2024-08-31T22:44:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a8cb31317a 2024-08-31T22:44:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:44:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:44:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:44:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:44:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:37:06 2024 2024-08-31T22:44:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:37:06 2024 2024-08-31T22:44:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:38:07 2024 2024-08-31T22:44:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:44:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a8cb31317a 2024-08-31T22:44:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:44:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:44:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 20ca808a-4ef8-4b55-a690-9ae6acbec89a 2024-08-31T22:44:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:44:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_31 snap 2024-08-31T22:44:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_31 2024-08-31T22:44:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:44:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_31.export 2024-08-31T22:44:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_31.export 2024-08-31T22:44:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_31.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_31.export 2024-08-31T22:44:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_31.export 2024-08-31T22:44:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_31.export 2024-08-31T22:44:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_31.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_31.export 2024-08-31T22:44:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_31.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_31.export 2024-08-31T22:44:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:44:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_32 2024-08-31T22:44:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_32 snap 2024-08-31T22:44:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_32 2024-08-31T22:44:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:44:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-08-31T22:44:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_32 2024-08-31T22:44:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-08-31T22:44:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_32 2024-08-31T22:44:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:44:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:26.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-08-31T22:44:26.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:26.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_32 2024-08-31T22:44:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:44:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:44:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:44:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-08-31T22:44:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:44:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:44:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:44:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:44:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:44:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:44:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-08-31T22:44:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:44:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:44:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:44:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:44:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:44:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:44:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:44:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:44:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:44:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:44:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:44:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:44:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:44:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-08-31T22:44:26.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-08-31T22:44:26.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:44:26.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:44:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2024-08-31T22:44:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:44:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:44:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:44:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:44:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:44:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:44:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:44:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:44:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:44:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:44:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:44:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:44:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:44:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:44:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:44:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:44:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:44:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:44:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:44:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:44:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:44:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:44:26.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:26.579 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2024-08-31T22:44:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:44:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:44:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:44:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:44:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:44:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:44:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:44:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-08-31T22:44:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_32 2024-08-31T22:44:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:44:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_32 2024-08-31T22:44:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_32 2024-08-31T22:44:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:44:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:44:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:44:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:44:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:44:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_32 2024-08-31T22:44:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_32 2024-08-31T22:44:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:44:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_32 ']' 2024-08-31T22:44:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_32' 2024-08-31T22:44:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2024-08-31T22:44:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:44:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:44:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:44:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:44:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:44:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2024-08-31T22:44:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:44:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:44:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:44:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:44:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:44:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:44:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:44:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:44:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:44:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:44:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:44:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:44:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:44:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2024-08-31T22:44:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-08-31T22:44:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:44:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:44:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_32 2024-08-31T22:44:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:44:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:44:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:44:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:44:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:44:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:44:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:44:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:44:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:44:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:44:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:44:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:44:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:44:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:44:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:44:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:44:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:44:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:44:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:44:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:44:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:44:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:44:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_32 2024-08-31T22:44:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_32 2024-08-31T22:44:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_32 2024-08-31T22:44:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_32 '' 2024-08-31T22:44:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_32 2024-08-31T22:44:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:44:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-08-31T22:44:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_32.status 2024-08-31T22:44:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_32.status 2024-08-31T22:44:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-08-31T22:44:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_32.status 2024-08-31T22:44:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:44:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:44:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5242], [object_number=7, tag_tid=1, entry_tid=5239]]], state=connected] 2024-08-31T22:44:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5242], [object_number=7, tag_tid=1, entry_tid=5239]]], state=connected] 2024-08-31T22:44:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_32.status 2024-08-31T22:44:27.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:44:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2024-08-31T22:44:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_32 2024-08-31T22:44:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2024-08-31T22:44:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_32 2024-08-31T22:44:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:44:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-08-31T22:44:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_32.status 2024-08-31T22:44:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_32.status 2024-08-31T22:44:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-08-31T22:44:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_32.status 2024-08-31T22:44:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:44:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:44:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5242], [object_number=7, tag_tid=1, entry_tid=5239]]], state=connected] 2024-08-31T22:44:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5242], [object_number=7, tag_tid=1, entry_tid=5239]]], state=connected] 2024-08-31T22:44:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_32.status 2024-08-31T22:44:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:44:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:44:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-08-31T22:44:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_32 2024-08-31T22:44:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:44:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-08-31T22:44:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_32': 2024-08-31T22:44:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:44:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:44:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:44:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10abd6782c89 2024-08-31T22:44:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10abd6782c89 2024-08-31T22:44:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:44:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:44:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:44:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:44:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:37:17 2024 2024-08-31T22:44:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:37:17 2024 2024-08-31T22:44:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:38:17 2024 2024-08-31T22:44:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:44:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10abd6782c89 2024-08-31T22:44:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:44:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:44:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 54fb8299-0e75-4c64-861f-a1f78fb4302f 2024-08-31T22:44:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:44:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_32 snap 2024-08-31T22:44:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_32 2024-08-31T22:44:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:44:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_32.export 2024-08-31T22:44:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_32.export 2024-08-31T22:44:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_32.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_32.export 2024-08-31T22:44:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_32.export 2024-08-31T22:44:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_32.export 2024-08-31T22:44:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_32.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_32.export 2024-08-31T22:44:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_32.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_32.export 2024-08-31T22:44:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:44:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_33 2024-08-31T22:44:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_33 snap 2024-08-31T22:44:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_33 2024-08-31T22:44:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:44:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-08-31T22:44:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_33 2024-08-31T22:44:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-08-31T22:44:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_33 2024-08-31T22:44:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:44:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-08-31T22:44:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_33 2024-08-31T22:44:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:44:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:44:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:44:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-08-31T22:44:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:44:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:32.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:32.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:44:32.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:44:32.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:44:32.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:32.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:44:32.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:32.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:44:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-08-31T22:44:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:44:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:44:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:44:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:44:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:44:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:44:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:44:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:44:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:44:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:44:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:44:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:44:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:44:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-08-31T22:44:32.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:44:32.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:44:32.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:44:32.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:44:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:44:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:44:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:44:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-08-31T22:44:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_33 2024-08-31T22:44:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:44:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_33 2024-08-31T22:44:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_33 2024-08-31T22:44:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:44:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:44:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:44:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:44:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:44:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_33 2024-08-31T22:44:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_33 2024-08-31T22:44:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:44:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_33 ']' 2024-08-31T22:44:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-08-31T22:44:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-08-31T22:44:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:44:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:44:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:44:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:44:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:44:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-08-31T22:44:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:44:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:44:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:44:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:44:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:44:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:44:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:44:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:44:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:44:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:44:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:44:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:44:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:44:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-08-31T22:44:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-08-31T22:44:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_33 2024-08-31T22:44:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-08-31T22:44:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_33 2024-08-31T22:44:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:44:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-08-31T22:44:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_33.status 2024-08-31T22:44:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_33.status 2024-08-31T22:44:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-08-31T22:44:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_33.status 2024-08-31T22:44:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:44:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:44:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12411], [object_number=14, tag_tid=1, entry_tid=12410]]], state=connected] 2024-08-31T22:44:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12411], [object_number=14, tag_tid=1, entry_tid=12410]]], state=connected] 2024-08-31T22:44:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_33.status 2024-08-31T22:44:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-08-31T22:44:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_33 2024-08-31T22:44:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-08-31T22:44:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_33 2024-08-31T22:44:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:44:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-08-31T22:44:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_33.status 2024-08-31T22:44:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_33.status 2024-08-31T22:44:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-08-31T22:44:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_33.status 2024-08-31T22:44:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:44:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:44:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12411], [object_number=14, tag_tid=1, entry_tid=12410]]], state=connected] 2024-08-31T22:44:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12411], [object_number=14, tag_tid=1, entry_tid=12410]]], state=connected] 2024-08-31T22:44:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_33.status 2024-08-31T22:44:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-08-31T22:44:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_33 2024-08-31T22:44:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:44:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-08-31T22:44:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_33': 2024-08-31T22:44:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:44:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:44:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:44:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a82a1d6073 2024-08-31T22:44:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a82a1d6073 2024-08-31T22:44:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:44:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:44:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:44:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:44:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:37:22 2024 2024-08-31T22:44:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:37:22 2024 2024-08-31T22:44:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:38:23 2024 2024-08-31T22:44:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:44:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a82a1d6073 2024-08-31T22:44:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:44:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:44:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 07b6bc0f-e63f-45c2-9304-88134d3c16d8 2024-08-31T22:44:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:44:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_33 snap 2024-08-31T22:44:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_33 2024-08-31T22:44:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:44:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_33.export 2024-08-31T22:44:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_33.export 2024-08-31T22:44:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_33.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_33.export 2024-08-31T22:44:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_33.export 2024-08-31T22:44:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_33.export 2024-08-31T22:44:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_33.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_33.export 2024-08-31T22:44:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_33.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_33.export 2024-08-31T22:44:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:44:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_34 2024-08-31T22:44:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_34 snap 2024-08-31T22:44:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_34 2024-08-31T22:44:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:44:35.417 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:44:35.415+0000 7f13efcb9640 -1 reset not still connected to 0x559f0fd4dd40 2024-08-31T22:44:35.418 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:44:35.415+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff16b60 2024-08-31T22:44:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-08-31T22:44:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_34 2024-08-31T22:44:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-08-31T22:44:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_34 2024-08-31T22:44:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:44:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-08-31T22:44:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_34 2024-08-31T22:44:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:44:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:44:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:44:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-08-31T22:44:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:44:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:44:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:44:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:44:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:44:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:44:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-08-31T22:44:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:44:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:44:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:44:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:44:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:44:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:44:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:44:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:44:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:44:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:44:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:44:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:44:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:44:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-08-31T22:44:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-08-31T22:44:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:44:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:44:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2024-08-31T22:44:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:44:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:44:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:44:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:44:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:44:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:44:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:44:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:44:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:44:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:44:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:44:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:44:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:44:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:44:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:44:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:44:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:44:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:44:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:44:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:44:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:44:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:44:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2024-08-31T22:44:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:44:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:44:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:44:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:44:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:44:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:44:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:44:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-08-31T22:44:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_34 2024-08-31T22:44:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:44:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_34 2024-08-31T22:44:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_34 2024-08-31T22:44:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:44:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:44:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:44:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:44:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:44:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_34 2024-08-31T22:44:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_34 2024-08-31T22:44:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:44:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_34 ']' 2024-08-31T22:44:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_34' 2024-08-31T22:44:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2024-08-31T22:44:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:44:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:44:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:44:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:44:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:44:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2024-08-31T22:44:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:44:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:44:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:44:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:44:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:44:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:44:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:44:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:44:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:44:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:44:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:44:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:44:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:44:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2024-08-31T22:44:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-08-31T22:44:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:44:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:44:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2024-08-31T22:44:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:44:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:44:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:44:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:44:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:44:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:44:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:44:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:44:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:44:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:44:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:44:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:44:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:44:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:44:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:44:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:44:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:44:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:44:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:44:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:44:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:44:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:44:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2024-08-31T22:44:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_34 2024-08-31T22:44:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_34 2024-08-31T22:44:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_34 '' 2024-08-31T22:44:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_34 2024-08-31T22:44:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:44:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-08-31T22:44:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_34.status 2024-08-31T22:44:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_34.status 2024-08-31T22:44:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-08-31T22:44:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_34.status 2024-08-31T22:44:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:44:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:44:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10655], [object_number=14, tag_tid=1, entry_tid=10654]]], state=connected] 2024-08-31T22:44:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10655], [object_number=14, tag_tid=1, entry_tid=10654]]], state=connected] 2024-08-31T22:44:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_34.status 2024-08-31T22:44:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2024-08-31T22:44:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_34 2024-08-31T22:44:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2024-08-31T22:44:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_34 2024-08-31T22:44:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:44:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-08-31T22:44:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_34.status 2024-08-31T22:44:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_34.status 2024-08-31T22:44:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-08-31T22:44:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_34.status 2024-08-31T22:44:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:44:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:44:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10655], [object_number=14, tag_tid=1, entry_tid=10654]]], state=connected] 2024-08-31T22:44:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10655], [object_number=14, tag_tid=1, entry_tid=10654]]], state=connected] 2024-08-31T22:44:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_34.status 2024-08-31T22:44:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-08-31T22:44:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_34 2024-08-31T22:44:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:44:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-08-31T22:44:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_34': 2024-08-31T22:44:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:44:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:44:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:44:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10ab8a1d413 2024-08-31T22:44:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10ab8a1d413 2024-08-31T22:44:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:44:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:44:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:44:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:44:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:37:37 2024 2024-08-31T22:44:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:37:37 2024 2024-08-31T22:44:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:38:38 2024 2024-08-31T22:44:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:44:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10ab8a1d413 2024-08-31T22:44:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:44:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:44:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 26bf9c3d-a2cf-48d6-aec7-8b455facdf1b 2024-08-31T22:44:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:44:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_34 snap 2024-08-31T22:44:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_34 2024-08-31T22:44:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:44:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_34.export 2024-08-31T22:44:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_34.export 2024-08-31T22:44:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_34.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_34.export 2024-08-31T22:44:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_34.export 2024-08-31T22:44:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_34.export 2024-08-31T22:44:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_34.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_34.export 2024-08-31T22:44:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_34.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_34.export 2024-08-31T22:44:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:44:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_35 2024-08-31T22:44:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_35 snap 2024-08-31T22:44:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_35 2024-08-31T22:44:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:44:41.466 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:44:41.462+0000 7ff273ffe640 -1 reset not still connected to 0x56194cb5b520 2024-08-31T22:44:41.466 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:44:41.462+0000 7ff273ffe640 -1 reset not still connected to 0x56194e533ad0 2024-08-31T22:44:41.466 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:44:41.462+0000 7ff273ffe640 -1 reset not still connected to 0x5619515f5110 2024-08-31T22:44:41.466 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:44:41.462+0000 7ff273ffe640 -1 reset not still connected to 0x561954ef31e0 2024-08-31T22:44:41.466 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:44:41.462+0000 7ff273ffe640 -1 reset not still connected to 0x56195ab93ee0 2024-08-31T22:44:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-08-31T22:44:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_35 2024-08-31T22:44:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-08-31T22:44:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_35 2024-08-31T22:44:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:44:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-08-31T22:44:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_35 2024-08-31T22:44:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:44:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:44:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:44:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-08-31T22:44:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:44:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:44:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:44:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:44:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:44:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:44:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-08-31T22:44:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:44:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:44:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:44:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:44:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:44:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:44:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:44:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:44:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:44:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:44:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:44:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:44:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:44:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-08-31T22:44:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:44:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:44:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:44:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:44:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:44:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:44:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:44:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2024-08-31T22:44:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_35 2024-08-31T22:44:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:44:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_35 2024-08-31T22:44:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_35 2024-08-31T22:44:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:44:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:44:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:44:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:44:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:44:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_35 2024-08-31T22:44:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_35 2024-08-31T22:44:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:44:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_35 ']' 2024-08-31T22:44:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-08-31T22:44:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-08-31T22:44:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:44:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:44:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:44:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:44:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:44:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-08-31T22:44:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:44:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:44:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:44:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:44:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:44:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:44:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:44:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:44:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:44:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:44:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:44:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:44:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:44:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-08-31T22:44:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-08-31T22:44:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_35 2024-08-31T22:44:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-08-31T22:44:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_35 2024-08-31T22:44:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:44:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-08-31T22:44:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_35.status 2024-08-31T22:44:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_35.status 2024-08-31T22:44:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-08-31T22:44:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_35.status 2024-08-31T22:44:44.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:44:44.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:44:44.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:44.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=11042], [object_number=15, tag_tid=1, entry_tid=11039]]], state=connected] 2024-08-31T22:44:44.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=11042], [object_number=15, tag_tid=1, entry_tid=11039]]], state=connected] 2024-08-31T22:44:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_35.status 2024-08-31T22:44:44.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-08-31T22:44:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_35 2024-08-31T22:44:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-08-31T22:44:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_35 2024-08-31T22:44:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:44:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-08-31T22:44:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_35.status 2024-08-31T22:44:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_35.status 2024-08-31T22:44:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-08-31T22:44:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_35.status 2024-08-31T22:44:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:44:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:44:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=11042], [object_number=15, tag_tid=1, entry_tid=11039]]], state=connected] 2024-08-31T22:44:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=11042], [object_number=15, tag_tid=1, entry_tid=11039]]], state=connected] 2024-08-31T22:44:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_35.status 2024-08-31T22:44:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-08-31T22:44:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_35 2024-08-31T22:44:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:44:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-08-31T22:44:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_35': 2024-08-31T22:44:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:44:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:44:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:44:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a81ece8f2b 2024-08-31T22:44:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a81ece8f2b 2024-08-31T22:44:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:44:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:44:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:44:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:44:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:37:50 2024 2024-08-31T22:44:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:37:50 2024 2024-08-31T22:44:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:38:51 2024 2024-08-31T22:44:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:44:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a81ece8f2b 2024-08-31T22:44:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:44:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:44:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 963f91a0-7562-4b89-b0da-5cc1c13b09b1 2024-08-31T22:44:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:44:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_35 snap 2024-08-31T22:44:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_35 2024-08-31T22:44:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:44:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_35.export 2024-08-31T22:44:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_35.export 2024-08-31T22:44:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_35.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_35.export 2024-08-31T22:44:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_35.export 2024-08-31T22:44:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_35.export 2024-08-31T22:44:45.392 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:44:45.394+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff161a0 2024-08-31T22:44:45.392 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:44:45.394+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff169c0 2024-08-31T22:44:45.392 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:44:45.394+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff16d00 2024-08-31T22:44:45.392 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:44:45.394+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff17c70 2024-08-31T22:44:45.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_35.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_35.export 2024-08-31T22:44:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_35.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_35.export 2024-08-31T22:44:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:44:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_36 2024-08-31T22:44:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_36 snap 2024-08-31T22:44:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_36 2024-08-31T22:44:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:44:46.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-08-31T22:44:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_36 2024-08-31T22:44:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-08-31T22:44:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_36 2024-08-31T22:44:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:44:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-08-31T22:44:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_36 2024-08-31T22:44:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:44:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:44:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:44:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-08-31T22:44:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:44:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:44:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:44:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:44:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:44:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:44:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-08-31T22:44:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:44:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:44:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:44:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:44:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:44:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:44:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:44:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:44:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:44:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:44:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:44:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:44:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:44:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-08-31T22:44:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-08-31T22:44:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:44:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:44:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-08-31T22:44:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:44:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:44:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:44:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:44:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:44:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:44:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:44:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:44:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:44:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:44:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:44:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:44:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:44:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:44:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:44:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:44:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:44:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:44:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:44:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:44:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:44:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:44:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-08-31T22:44:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:44:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:44:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:44:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:44:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:44:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:44:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:44:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2024-08-31T22:44:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_36 2024-08-31T22:44:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:44:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_36 2024-08-31T22:44:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_36 2024-08-31T22:44:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:44:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:44:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:44:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:44:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:44:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_36 2024-08-31T22:44:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_36 2024-08-31T22:44:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:44:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_36 ']' 2024-08-31T22:44:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_36' 2024-08-31T22:44:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2024-08-31T22:44:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:44:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:44:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:44:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:44:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:44:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2024-08-31T22:44:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:44:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:44:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:44:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:44:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:44:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:44:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:44:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:44:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:44:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:44:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:44:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:44:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:44:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2024-08-31T22:44:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-08-31T22:44:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:44:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:44:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_36 2024-08-31T22:44:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:44:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:44:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:44:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:44:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:44:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:44:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:44:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:44:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:44:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:44:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:44:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:44:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:44:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:44:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:44:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:44:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:44:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:44:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:44:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:44:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:44:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:44:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:44:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:44:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_36 2024-08-31T22:44:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_36 2024-08-31T22:44:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_36 2024-08-31T22:44:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_36 '' 2024-08-31T22:44:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_36 2024-08-31T22:44:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:44:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-08-31T22:44:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_36.status 2024-08-31T22:44:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_36.status 2024-08-31T22:44:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-08-31T22:44:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_36.status 2024-08-31T22:44:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:44:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:44:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6807], [object_number=6, tag_tid=1, entry_tid=6806]]], state=connected] 2024-08-31T22:44:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6807], [object_number=6, tag_tid=1, entry_tid=6806]]], state=connected] 2024-08-31T22:44:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_36.status 2024-08-31T22:44:48.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-08-31T22:44:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2024-08-31T22:44:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_36 2024-08-31T22:44:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2024-08-31T22:44:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_36 2024-08-31T22:44:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:44:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-08-31T22:44:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_36.status 2024-08-31T22:44:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:48.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_36.status 2024-08-31T22:44:48.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-08-31T22:44:48.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_36.status 2024-08-31T22:44:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:44:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:44:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6807], [object_number=6, tag_tid=1, entry_tid=6806]]], state=connected] 2024-08-31T22:44:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6807], [object_number=6, tag_tid=1, entry_tid=6806]]], state=connected] 2024-08-31T22:44:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_36.status 2024-08-31T22:44:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-08-31T22:44:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-08-31T22:44:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-08-31T22:44:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_36 2024-08-31T22:44:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:44:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-08-31T22:44:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_36': 2024-08-31T22:44:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:44:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:44:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:44:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10abdba9156 2024-08-31T22:44:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10abdba9156 2024-08-31T22:44:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:44:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:44:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:44:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:44:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:38:04 2024 2024-08-31T22:44:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:38:04 2024 2024-08-31T22:44:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:39:05 2024 2024-08-31T22:44:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:44:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10abdba9156 2024-08-31T22:44:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:44:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:44:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: db712ee2-9e18-4e62-a5cb-c897e571a527 2024-08-31T22:44:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:44:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_36 snap 2024-08-31T22:44:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_36 2024-08-31T22:44:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:44:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_36.export 2024-08-31T22:44:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_36.export 2024-08-31T22:44:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_36.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_36.export 2024-08-31T22:44:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_36.export 2024-08-31T22:44:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_36.export 2024-08-31T22:44:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_36.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_36.export 2024-08-31T22:44:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_36.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_36.export 2024-08-31T22:44:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:44:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_37 2024-08-31T22:44:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_37 snap 2024-08-31T22:44:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_37 2024-08-31T22:44:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:44:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-08-31T22:44:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_37 2024-08-31T22:44:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-08-31T22:44:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_37 2024-08-31T22:44:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:44:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-08-31T22:44:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_37 2024-08-31T22:44:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:44:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:44:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:44:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-08-31T22:44:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:44:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:44:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:44:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:44:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:44:53.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:53.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:44:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-08-31T22:44:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:44:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:44:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:44:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:44:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:44:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:44:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:44:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:44:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:44:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:44:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:44:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:44:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:44:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:44:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:44:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-08-31T22:44:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:44:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:44:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:44:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:44:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:44:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:44:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:44:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2024-08-31T22:44:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_37 2024-08-31T22:44:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:44:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_37 2024-08-31T22:44:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:44:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_37 2024-08-31T22:44:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:44:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:44:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:44:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:44:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:44:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_37 2024-08-31T22:44:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_37 2024-08-31T22:44:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:44:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_37 ']' 2024-08-31T22:44:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_37' 2024-08-31T22:44:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_37 2024-08-31T22:44:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:44:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:44:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:44:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:44:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:44:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_37 2024-08-31T22:44:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:44:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:44:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:44:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:44:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:44:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:44:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:44:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:44:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:44:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:44:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:44:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:44:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:44:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_37 2024-08-31T22:44:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_37 2024-08-31T22:44:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_37 2024-08-31T22:44:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_37 '' 2024-08-31T22:44:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_37 2024-08-31T22:44:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:44:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-08-31T22:44:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_37.status 2024-08-31T22:44:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_37.status 2024-08-31T22:44:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_37.status 2024-08-31T22:44:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-08-31T22:44:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:44:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:44:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12243], [object_number=14, tag_tid=1, entry_tid=12242]]], state=connected] 2024-08-31T22:44:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=12244], [object_number=15, tag_tid=1, entry_tid=12243], [object_number=14, tag_tid=1, entry_tid=12242], [object_number=13, tag_tid=1, entry_tid=12241]]], state=connected] 2024-08-31T22:44:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_37.status 2024-08-31T22:44:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_37 2024-08-31T22:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_37 2024-08-31T22:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_37 '..*' 2024-08-31T22:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_37 2024-08-31T22:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-08-31T22:44:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_37.status 2024-08-31T22:44:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_37.status 2024-08-31T22:44:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-08-31T22:44:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_37.status 2024-08-31T22:44:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:44:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:44:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12243], [object_number=14, tag_tid=1, entry_tid=12242]]], state=connected] 2024-08-31T22:44:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=12244], [object_number=15, tag_tid=1, entry_tid=12243], [object_number=14, tag_tid=1, entry_tid=12242], [object_number=13, tag_tid=1, entry_tid=12241]]], state=connected] 2024-08-31T22:44:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_37.status 2024-08-31T22:44:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=12, tag_tid=1, entry_tid=12244' 2024-08-31T22:44:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=12, tag_tid=1, entry_tid=12244' 2024-08-31T22:44:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=12, tag_tid=1, entry_tid=12244' '!=' '' 2024-08-31T22:44:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:44:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=12, tag_tid=1, entry_tid=12244' = '' 2024-08-31T22:44:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=12, tag_tid=1, entry_tid=12244' 2024-08-31T22:44:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:44:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:44:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=2 2024-08-31T22:44:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=12, tag_tid=1, entry_tid=12244' 2024-08-31T22:44:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:44:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:44:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=1 2024-08-31T22:44:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:44:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:44:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:44:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=12, tag_tid=1, entry_tid=12244' 2024-08-31T22:44:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:44:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:44:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=12244 2024-08-31T22:44:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 2 = 1 -a 1 -le 12244 2024-08-31T22:44:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:44:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:44:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:44:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_37 2024-08-31T22:44:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_37 2024-08-31T22:44:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:44:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_37 ']' 2024-08-31T22:44:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_37' 2024-08-31T22:44:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_37 2024-08-31T22:44:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:44:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:44:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:44:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:44:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:44:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:44:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:44:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_37 2024-08-31T22:44:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:44:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:44:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:44:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:44:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:44:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:44:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:44:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:44:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:44:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:44:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:44:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:54.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:44:54.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:44:54.214 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:44:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:44:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:44:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:44:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:44:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:44:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:44:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:44:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:44:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:44:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:44:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:44:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:44:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:44:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:54.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:54.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:44:54.241 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_37 2024-08-31T22:44:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_37 2024-08-31T22:44:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_37 2024-08-31T22:44:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_37 '' 2024-08-31T22:44:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_37 2024-08-31T22:44:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:44:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-08-31T22:44:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_37.status 2024-08-31T22:44:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_37.status 2024-08-31T22:44:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-08-31T22:44:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_37.status 2024-08-31T22:44:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:44:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:44:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12243], [object_number=14, tag_tid=1, entry_tid=12242]]], state=connected] 2024-08-31T22:44:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12243], [object_number=14, tag_tid=1, entry_tid=12242]]], state=connected] 2024-08-31T22:44:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_37.status 2024-08-31T22:44:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_37 2024-08-31T22:44:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_37 2024-08-31T22:44:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_37 '..*' 2024-08-31T22:44:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:44:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:44:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_37 2024-08-31T22:44:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:44:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-08-31T22:44:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_37.status 2024-08-31T22:44:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:44:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_37.status 2024-08-31T22:44:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-08-31T22:44:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_37.status 2024-08-31T22:44:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:44:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:44:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:44:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12243], [object_number=14, tag_tid=1, entry_tid=12242]]], state=connected] 2024-08-31T22:44:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12243], [object_number=14, tag_tid=1, entry_tid=12242]]], state=connected] 2024-08-31T22:44:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_37.status 2024-08-31T22:44:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:44:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-08-31T22:44:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_37 2024-08-31T22:44:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:44:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:44:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-08-31T22:44:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_37': 2024-08-31T22:44:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:44:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:44:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:44:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a8f3aa897f 2024-08-31T22:44:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a8f3aa897f 2024-08-31T22:44:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:44:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:44:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:44:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:44:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:38:11 2024 2024-08-31T22:44:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:38:11 2024 2024-08-31T22:44:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:39:12 2024 2024-08-31T22:44:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:44:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a8f3aa897f 2024-08-31T22:44:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:44:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:44:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: db95a88e-0e89-45af-955d-257c23f740ca 2024-08-31T22:44:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:44:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_37 snap 2024-08-31T22:44:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_37 2024-08-31T22:44:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:44:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_37.export 2024-08-31T22:44:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_37.export 2024-08-31T22:44:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_37.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_37.export 2024-08-31T22:44:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_37.export 2024-08-31T22:44:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_37.export 2024-08-31T22:44:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:44:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_37.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_37.export 2024-08-31T22:44:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_37.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_37.export 2024-08-31T22:44:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:44:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:44:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_38 2024-08-31T22:44:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_38 snap 2024-08-31T22:44:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:44:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_38 2024-08-31T22:44:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:44:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:44:59.265 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:44:59.266+0000 7f13efcb9640 -1 reset not still connected to 0x559f26ea96c0 2024-08-31T22:44:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-08-31T22:44:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_38 2024-08-31T22:44:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-08-31T22:44:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:44:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:44:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_38 2024-08-31T22:44:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:44:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:44:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:44:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:00.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-08-31T22:45:00.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_38 2024-08-31T22:45:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:45:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:45:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:45:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:45:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-08-31T22:45:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:45:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:45:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:45:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:45:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:45:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-08-31T22:45:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:45:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:45:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:45:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:45:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:45:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:45:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:45:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:45:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:45:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:45:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:45:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:45:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:45:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-08-31T22:45:00.404 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:45:00.404+0000 7f338ab71640 -1 reset not still connected to 0x555ce293a000 2024-08-31T22:45:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-08-31T22:45:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:45:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:45:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2024-08-31T22:45:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:45:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:45:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:45:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:45:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:45:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:45:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:45:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:45:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:45:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:45:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:45:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:45:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:45:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:45:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:45:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:45:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:45:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:45:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:45:00.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:45:00.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:45:00.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:45:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2024-08-31T22:45:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:45:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:45:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:45:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:45:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:45:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:45:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:45:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2024-08-31T22:45:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_38 2024-08-31T22:45:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:45:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_38 2024-08-31T22:45:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_38 2024-08-31T22:45:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:45:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:45:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:45:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:45:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:45:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_38 2024-08-31T22:45:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_38 2024-08-31T22:45:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:45:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_38 ']' 2024-08-31T22:45:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_38' 2024-08-31T22:45:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2024-08-31T22:45:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:45:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:45:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:45:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:45:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:45:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2024-08-31T22:45:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:45:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:45:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:45:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:45:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:45:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:45:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:45:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:45:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:45:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:45:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:45:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:45:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:45:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:00.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2024-08-31T22:45:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-08-31T22:45:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:45:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:45:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2024-08-31T22:45:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:45:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:45:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:45:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:45:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:45:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:45:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:45:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:45:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:45:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:45:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:45:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:45:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:45:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:45:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:45:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:45:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:45:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:45:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:45:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:45:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:45:01.074 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:45:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2024-08-31T22:45:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_38 2024-08-31T22:45:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_38 2024-08-31T22:45:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_38 '' 2024-08-31T22:45:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_38 2024-08-31T22:45:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:45:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-08-31T22:45:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_38.status 2024-08-31T22:45:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_38.status 2024-08-31T22:45:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-08-31T22:45:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_38.status 2024-08-31T22:45:01.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:45:01.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:45:01.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:01.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8966], [object_number=11, tag_tid=1, entry_tid=8963]]], state=connected] 2024-08-31T22:45:01.307 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8966], [object_number=11, tag_tid=1, entry_tid=8963]]], state=connected] 2024-08-31T22:45:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_38.status 2024-08-31T22:45:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2024-08-31T22:45:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_38 2024-08-31T22:45:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2024-08-31T22:45:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_38 2024-08-31T22:45:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:45:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-08-31T22:45:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_38.status 2024-08-31T22:45:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:01.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_38.status 2024-08-31T22:45:01.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-08-31T22:45:01.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_38.status 2024-08-31T22:45:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:45:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:45:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8966], [object_number=11, tag_tid=1, entry_tid=8963]]], state=connected] 2024-08-31T22:45:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8966], [object_number=11, tag_tid=1, entry_tid=8963]]], state=connected] 2024-08-31T22:45:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_38.status 2024-08-31T22:45:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-08-31T22:45:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_38 2024-08-31T22:45:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:45:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-08-31T22:45:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_38': 2024-08-31T22:45:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:45:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:45:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:45:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10abe9a03f72 2024-08-31T22:45:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10abe9a03f72 2024-08-31T22:45:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:45:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:45:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:45:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:45:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:38:27 2024 2024-08-31T22:45:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:38:27 2024 2024-08-31T22:45:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:39:27 2024 2024-08-31T22:45:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:45:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10abe9a03f72 2024-08-31T22:45:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:45:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:45:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: b7fb886a-e7d4-4ff8-9f15-b3d4f1571bca 2024-08-31T22:45:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:45:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_38 snap 2024-08-31T22:45:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_38 2024-08-31T22:45:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:45:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_38.export 2024-08-31T22:45:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_38.export 2024-08-31T22:45:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_38.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_38.export 2024-08-31T22:45:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_38.export 2024-08-31T22:45:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_38.export 2024-08-31T22:45:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_38.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_38.export 2024-08-31T22:45:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_38.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_38.export 2024-08-31T22:45:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:45:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_39 2024-08-31T22:45:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_39 snap 2024-08-31T22:45:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_39 2024-08-31T22:45:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:45:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-08-31T22:45:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_39 2024-08-31T22:45:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-08-31T22:45:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_39 2024-08-31T22:45:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:45:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-08-31T22:45:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_39 2024-08-31T22:45:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:45:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:45:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:45:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-08-31T22:45:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:45:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:45:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:45:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:45:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:45:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:45:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-08-31T22:45:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:45:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:45:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:45:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:45:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:45:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:45:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:45:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:45:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:45:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:45:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:45:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:45:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:45:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-08-31T22:45:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:45:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:45:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:45:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:45:06.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:45:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:45:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:45:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2024-08-31T22:45:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_39 2024-08-31T22:45:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:45:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_39 2024-08-31T22:45:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_39 2024-08-31T22:45:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:45:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:45:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:45:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:45:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:45:06.723 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:45:06.720+0000 7f6588f42640 -1 reset not still connected to 0x56202997c9c0 2024-08-31T22:45:06.727 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:45:06.724+0000 7f6588f42640 -1 reset not still connected to 0x562032a40680 2024-08-31T22:45:06.738 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:45:06.736+0000 7f6588f42640 -1 reset not still connected to 0x56204072d5f0 2024-08-31T22:45:06.739 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:45:06.736+0000 7f6588f42640 -1 reset not still connected to 0x562054723e10 2024-08-31T22:45:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_39 2024-08-31T22:45:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_39 2024-08-31T22:45:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:45:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_39 ']' 2024-08-31T22:45:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_39' 2024-08-31T22:45:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_39 2024-08-31T22:45:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:45:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:45:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:45:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:45:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:45:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_39 2024-08-31T22:45:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:45:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:45:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:45:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:45:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:45:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:45:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:45:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:45:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:45:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:45:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:45:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:45:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:45:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_39 2024-08-31T22:45:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_39 2024-08-31T22:45:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_39 2024-08-31T22:45:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_39 '' 2024-08-31T22:45:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_39 2024-08-31T22:45:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:45:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-08-31T22:45:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_39.status 2024-08-31T22:45:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_39.status 2024-08-31T22:45:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-08-31T22:45:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_39.status 2024-08-31T22:45:06.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:45:06.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:45:06.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7983], [object_number=10, tag_tid=1, entry_tid=7982]]], state=connected] 2024-08-31T22:45:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7983], [object_number=10, tag_tid=1, entry_tid=7982]]], state=connected] 2024-08-31T22:45:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_39.status 2024-08-31T22:45:07.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_39 2024-08-31T22:45:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_39 2024-08-31T22:45:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_39 '..*' 2024-08-31T22:45:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_39 2024-08-31T22:45:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:45:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-08-31T22:45:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_39.status 2024-08-31T22:45:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_39.status 2024-08-31T22:45:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-08-31T22:45:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_39.status 2024-08-31T22:45:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:45:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:45:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7983], [object_number=10, tag_tid=1, entry_tid=7982]]], state=connected] 2024-08-31T22:45:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7983], [object_number=10, tag_tid=1, entry_tid=7982]]], state=connected] 2024-08-31T22:45:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_39.status 2024-08-31T22:45:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-08-31T22:45:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_39 2024-08-31T22:45:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:07.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:45:07.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-08-31T22:45:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_39': 2024-08-31T22:45:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:45:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:45:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:45:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a834e68661 2024-08-31T22:45:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a834e68661 2024-08-31T22:45:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:45:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:45:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:45:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:45:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:38:38 2024 2024-08-31T22:45:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:38:38 2024 2024-08-31T22:45:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:39:38 2024 2024-08-31T22:45:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:45:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a834e68661 2024-08-31T22:45:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:45:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:45:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: ee3e6659-3f5d-4cf5-af65-76b48d7740b7 2024-08-31T22:45:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:45:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_39 snap 2024-08-31T22:45:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_39 2024-08-31T22:45:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:45:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_39.export 2024-08-31T22:45:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_39.export 2024-08-31T22:45:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_39.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_39.export 2024-08-31T22:45:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_39.export 2024-08-31T22:45:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_39.export 2024-08-31T22:45:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_39.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_39.export 2024-08-31T22:45:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_39.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_39.export 2024-08-31T22:45:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:45:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_40 2024-08-31T22:45:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_40 snap 2024-08-31T22:45:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_40 2024-08-31T22:45:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:45:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-08-31T22:45:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_40 2024-08-31T22:45:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-08-31T22:45:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_40 2024-08-31T22:45:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:45:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-08-31T22:45:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_40 2024-08-31T22:45:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:45:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:45:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:45:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-08-31T22:45:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:45:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:45:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:45:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:45:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:45:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:45:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-08-31T22:45:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:45:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:45:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:45:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:45:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:45:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:45:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:45:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:45:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:45:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:45:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:45:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:45:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:45:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-08-31T22:45:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-08-31T22:45:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:45:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:45:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2024-08-31T22:45:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:45:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:45:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:45:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:45:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:45:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:45:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:45:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:45:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:45:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:45:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:45:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:45:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:45:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:45:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:45:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:45:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:45:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:45:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:45:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:45:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:45:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:45:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2024-08-31T22:45:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:45:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:45:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:45:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:45:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:45:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:45:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:45:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2024-08-31T22:45:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_40 2024-08-31T22:45:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:45:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_40 2024-08-31T22:45:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_40 2024-08-31T22:45:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:45:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:45:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:45:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:45:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:45:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_40 2024-08-31T22:45:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_40 2024-08-31T22:45:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:45:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_40 ']' 2024-08-31T22:45:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_40' 2024-08-31T22:45:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2024-08-31T22:45:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:45:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:45:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:45:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:45:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:45:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2024-08-31T22:45:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:45:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:45:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:45:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:45:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:45:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:45:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:45:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:45:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:45:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:45:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:45:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:45:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:45:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:11.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:11.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:11.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:11.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2024-08-31T22:45:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-08-31T22:45:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:45:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:45:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_40 2024-08-31T22:45:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:45:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:45:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:45:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:45:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:45:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:45:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:45:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:45:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:45:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:45:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:45:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:45:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:45:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:45:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:45:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:45:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:45:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:45:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:45:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:45:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:45:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:45:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_40 2024-08-31T22:45:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_40 2024-08-31T22:45:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_40 2024-08-31T22:45:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_40 '' 2024-08-31T22:45:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_40 2024-08-31T22:45:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:45:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-08-31T22:45:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_40.status 2024-08-31T22:45:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_40.status 2024-08-31T22:45:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-08-31T22:45:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_40.status 2024-08-31T22:45:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:45:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:45:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11231], [object_number=14, tag_tid=1, entry_tid=11230]]], state=connected] 2024-08-31T22:45:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11231], [object_number=14, tag_tid=1, entry_tid=11230]]], state=connected] 2024-08-31T22:45:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_40.status 2024-08-31T22:45:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:45:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2024-08-31T22:45:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_40 2024-08-31T22:45:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2024-08-31T22:45:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_40 2024-08-31T22:45:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:45:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-08-31T22:45:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_40.status 2024-08-31T22:45:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_40.status 2024-08-31T22:45:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-08-31T22:45:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_40.status 2024-08-31T22:45:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:45:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:45:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11231], [object_number=14, tag_tid=1, entry_tid=11230]]], state=connected] 2024-08-31T22:45:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11231], [object_number=14, tag_tid=1, entry_tid=11230]]], state=connected] 2024-08-31T22:45:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_40.status 2024-08-31T22:45:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:45:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:45:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-08-31T22:45:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_40 2024-08-31T22:45:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:45:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-08-31T22:45:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_40': 2024-08-31T22:45:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:45:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:45:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:45:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10abd8b3aba 2024-08-31T22:45:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10abd8b3aba 2024-08-31T22:45:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:45:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:45:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:45:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:45:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:38:46 2024 2024-08-31T22:45:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:38:46 2024 2024-08-31T22:45:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:39:47 2024 2024-08-31T22:45:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:45:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10abd8b3aba 2024-08-31T22:45:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:45:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:45:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: be5eb171-17f7-470c-a869-3782cee10377 2024-08-31T22:45:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:45:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_40 snap 2024-08-31T22:45:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_40 2024-08-31T22:45:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:13.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:45:13.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_40.export 2024-08-31T22:45:13.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_40.export 2024-08-31T22:45:13.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_40.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_40.export 2024-08-31T22:45:13.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_40.export 2024-08-31T22:45:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_40.export 2024-08-31T22:45:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_40.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_40.export 2024-08-31T22:45:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_40.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_40.export 2024-08-31T22:45:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:45:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_41 2024-08-31T22:45:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_41 snap 2024-08-31T22:45:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_41 2024-08-31T22:45:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:45:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-08-31T22:45:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_41 2024-08-31T22:45:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-08-31T22:45:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_41 2024-08-31T22:45:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:45:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-08-31T22:45:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_41 2024-08-31T22:45:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:45:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:45:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:45:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-08-31T22:45:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:45:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:45:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:45:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:45:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:45:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:45:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-08-31T22:45:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:45:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:45:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:45:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:45:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:45:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:45:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:45:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:45:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:45:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:45:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:45:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:45:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:45:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-08-31T22:45:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:45:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:45:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:45:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:45:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:45:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:45:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:45:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2024-08-31T22:45:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_41 2024-08-31T22:45:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:45:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_41 2024-08-31T22:45:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_41 2024-08-31T22:45:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:45:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:45:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:45:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:45:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:45:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_41 2024-08-31T22:45:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_41 2024-08-31T22:45:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:45:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_41 ']' 2024-08-31T22:45:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_41' 2024-08-31T22:45:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_41 2024-08-31T22:45:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:45:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:45:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:45:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:45:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:45:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_41 2024-08-31T22:45:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:45:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:45:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:45:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:45:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:45:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:45:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:45:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:45:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:45:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:45:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:45:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:45:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:45:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:16.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:16.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:16.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:16.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_41 2024-08-31T22:45:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_41 2024-08-31T22:45:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_41 2024-08-31T22:45:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_41 '' 2024-08-31T22:45:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_41 2024-08-31T22:45:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:45:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-08-31T22:45:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_41.status 2024-08-31T22:45:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_41.status 2024-08-31T22:45:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-08-31T22:45:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_41.status 2024-08-31T22:45:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:45:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:45:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9739], [object_number=10, tag_tid=1, entry_tid=9738]]], state=connected] 2024-08-31T22:45:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9739], [object_number=10, tag_tid=1, entry_tid=9738]]], state=connected] 2024-08-31T22:45:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_41.status 2024-08-31T22:45:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_41 2024-08-31T22:45:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_41 2024-08-31T22:45:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_41 '..*' 2024-08-31T22:45:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_41 2024-08-31T22:45:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:45:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-08-31T22:45:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_41.status 2024-08-31T22:45:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_41.status 2024-08-31T22:45:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-08-31T22:45:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_41.status 2024-08-31T22:45:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:45:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:45:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9739], [object_number=10, tag_tid=1, entry_tid=9738]]], state=connected] 2024-08-31T22:45:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9739], [object_number=10, tag_tid=1, entry_tid=9738]]], state=connected] 2024-08-31T22:45:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_41.status 2024-08-31T22:45:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-08-31T22:45:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_41 2024-08-31T22:45:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:45:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-08-31T22:45:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_41': 2024-08-31T22:45:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:45:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:45:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:45:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a8a1734e9e 2024-08-31T22:45:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a8a1734e9e 2024-08-31T22:45:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:45:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:45:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:45:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:45:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:38:59 2024 2024-08-31T22:45:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:38:59 2024 2024-08-31T22:45:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:40:00 2024 2024-08-31T22:45:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:45:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a8a1734e9e 2024-08-31T22:45:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:45:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:45:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 6cb234f6-4181-4ddd-976e-1a8cb2a2c15f 2024-08-31T22:45:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:45:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_41 snap 2024-08-31T22:45:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_41 2024-08-31T22:45:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:45:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_41.export 2024-08-31T22:45:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_41.export 2024-08-31T22:45:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_41.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_41.export 2024-08-31T22:45:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_41.export 2024-08-31T22:45:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_41.export 2024-08-31T22:45:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_41.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_41.export 2024-08-31T22:45:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_41.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_41.export 2024-08-31T22:45:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:45:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_42 2024-08-31T22:45:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_42 snap 2024-08-31T22:45:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_42 2024-08-31T22:45:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:45:20.393 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:45:20.393+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff16b60 2024-08-31T22:45:20.410 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:45:20.413+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff16b60 2024-08-31T22:45:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-08-31T22:45:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_42 2024-08-31T22:45:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-08-31T22:45:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_42 2024-08-31T22:45:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:45:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-08-31T22:45:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_42 2024-08-31T22:45:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:45:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:45:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:45:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-08-31T22:45:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:45:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:45:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:45:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:45:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:45:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:45:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-08-31T22:45:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:45:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:45:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:45:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:45:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:45:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:45:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:45:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:45:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:45:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:45:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:45:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:45:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:45:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-08-31T22:45:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-08-31T22:45:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:45:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:45:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2024-08-31T22:45:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:45:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:45:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:45:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:45:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:45:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:45:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:45:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:45:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:45:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:45:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:45:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:45:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:45:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:45:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:45:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:45:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:45:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:45:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:45:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:45:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:45:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:45:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2024-08-31T22:45:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:45:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:45:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:45:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:45:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:45:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:45:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:45:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2024-08-31T22:45:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_42 2024-08-31T22:45:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:45:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_42 2024-08-31T22:45:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_42 2024-08-31T22:45:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:45:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:45:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:45:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:45:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:45:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_42 2024-08-31T22:45:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_42 2024-08-31T22:45:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:45:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_42 ']' 2024-08-31T22:45:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_42' 2024-08-31T22:45:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2024-08-31T22:45:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:45:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:45:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:45:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:45:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:45:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2024-08-31T22:45:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:45:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:45:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:45:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:45:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:45:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:45:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:45:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:45:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:45:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:45:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:45:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:45:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:45:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2024-08-31T22:45:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-08-31T22:45:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:45:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:45:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_42 2024-08-31T22:45:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:45:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:45:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:45:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:45:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:45:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:45:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:45:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:45:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:45:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:45:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:45:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:45:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:45:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:45:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:45:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:45:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:45:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:45:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:45:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:45:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:45:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:45:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_42 2024-08-31T22:45:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_42 2024-08-31T22:45:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_42 2024-08-31T22:45:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_42 '' 2024-08-31T22:45:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_42 2024-08-31T22:45:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:45:23.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-08-31T22:45:23.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_42.status 2024-08-31T22:45:23.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_42.status 2024-08-31T22:45:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-08-31T22:45:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_42.status 2024-08-31T22:45:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:45:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:45:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9671], [object_number=10, tag_tid=1, entry_tid=9670]]], state=connected] 2024-08-31T22:45:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9671], [object_number=10, tag_tid=1, entry_tid=9670]]], state=connected] 2024-08-31T22:45:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_42.status 2024-08-31T22:45:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2024-08-31T22:45:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_42 2024-08-31T22:45:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2024-08-31T22:45:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_42 2024-08-31T22:45:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:45:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-08-31T22:45:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_42.status 2024-08-31T22:45:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_42.status 2024-08-31T22:45:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-08-31T22:45:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_42.status 2024-08-31T22:45:23.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:45:23.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:45:23.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:23.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9671], [object_number=10, tag_tid=1, entry_tid=9670]]], state=connected] 2024-08-31T22:45:23.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9671], [object_number=10, tag_tid=1, entry_tid=9670]]], state=connected] 2024-08-31T22:45:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_42.status 2024-08-31T22:45:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-08-31T22:45:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_42 2024-08-31T22:45:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:45:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-08-31T22:45:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_42': 2024-08-31T22:45:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:45:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:45:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:45:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10ab3fd178a6 2024-08-31T22:45:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10ab3fd178a6 2024-08-31T22:45:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:45:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:45:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:45:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:45:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:39:12 2024 2024-08-31T22:45:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:39:12 2024 2024-08-31T22:45:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:40:13 2024 2024-08-31T22:45:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:45:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10ab3fd178a6 2024-08-31T22:45:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:45:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:45:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 0c60c3d4-fb56-4792-9b84-46351bef3e9a 2024-08-31T22:45:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:45:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_42 snap 2024-08-31T22:45:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_42 2024-08-31T22:45:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:45:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_42.export 2024-08-31T22:45:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_42.export 2024-08-31T22:45:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_42.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_42.export 2024-08-31T22:45:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_42.export 2024-08-31T22:45:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_42.export 2024-08-31T22:45:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_42.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_42.export 2024-08-31T22:45:25.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_42.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_42.export 2024-08-31T22:45:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:45:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_43 2024-08-31T22:45:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_43 snap 2024-08-31T22:45:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_43 2024-08-31T22:45:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:45:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-08-31T22:45:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_43 2024-08-31T22:45:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-08-31T22:45:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_43 2024-08-31T22:45:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:45:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-08-31T22:45:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_43 2024-08-31T22:45:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:45:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:45:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:45:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-08-31T22:45:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:45:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:45:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:45:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:45:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:45:28.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:28.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:45:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-08-31T22:45:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:45:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:45:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:45:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:45:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:45:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:45:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:45:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:45:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:45:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:45:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:45:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:45:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:45:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-08-31T22:45:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:45:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:45:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:45:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:45:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:45:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:45:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:45:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2024-08-31T22:45:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_43 2024-08-31T22:45:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:45:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_43 2024-08-31T22:45:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_43 2024-08-31T22:45:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:45:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:45:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:45:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:45:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:45:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_43 2024-08-31T22:45:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_43 2024-08-31T22:45:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:45:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_43 ']' 2024-08-31T22:45:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_43' 2024-08-31T22:45:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_43 2024-08-31T22:45:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:45:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:45:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:45:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:45:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:45:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_43 2024-08-31T22:45:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:45:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:45:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:45:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:45:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:45:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:45:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:45:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:45:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:45:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:45:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:45:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:45:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:45:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_43 2024-08-31T22:45:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_43 2024-08-31T22:45:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_43 2024-08-31T22:45:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_43 '' 2024-08-31T22:45:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_43 2024-08-31T22:45:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:45:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-08-31T22:45:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_43.status 2024-08-31T22:45:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_43.status 2024-08-31T22:45:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-08-31T22:45:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_43.status 2024-08-31T22:45:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:45:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:45:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7682], [object_number=11, tag_tid=1, entry_tid=7679]]], state=connected] 2024-08-31T22:45:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=7682], [object_number=9, tag_tid=1, entry_tid=7681], [object_number=8, tag_tid=1, entry_tid=7680], [object_number=11, tag_tid=1, entry_tid=7679]]], state=connected] 2024-08-31T22:45:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_43.status 2024-08-31T22:45:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_43 2024-08-31T22:45:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_43 2024-08-31T22:45:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_43 '..*' 2024-08-31T22:45:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_43 2024-08-31T22:45:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:45:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-08-31T22:45:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_43.status 2024-08-31T22:45:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_43.status 2024-08-31T22:45:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-08-31T22:45:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_43.status 2024-08-31T22:45:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:45:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:45:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7682], [object_number=11, tag_tid=1, entry_tid=7679]]], state=connected] 2024-08-31T22:45:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=7682], [object_number=9, tag_tid=1, entry_tid=7681], [object_number=8, tag_tid=1, entry_tid=7680], [object_number=11, tag_tid=1, entry_tid=7679]]], state=connected] 2024-08-31T22:45:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_43.status 2024-08-31T22:45:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=7682' 2024-08-31T22:45:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=7682' 2024-08-31T22:45:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=7682' '!=' '' 2024-08-31T22:45:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:45:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=7682' = '' 2024-08-31T22:45:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=10, tag_tid=1, entry_tid=7682' 2024-08-31T22:45:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:45:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:45:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=2 2024-08-31T22:45:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=10, tag_tid=1, entry_tid=7682' 2024-08-31T22:45:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:45:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:45:29.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=1 2024-08-31T22:45:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:45:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:45:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:45:29.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=10, tag_tid=1, entry_tid=7682' 2024-08-31T22:45:29.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:45:29.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:45:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=7682 2024-08-31T22:45:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 2 = 1 -a 1 -le 7682 2024-08-31T22:45:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:45:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:45:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:45:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_43 2024-08-31T22:45:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_43 2024-08-31T22:45:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:45:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_43 ']' 2024-08-31T22:45:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_43' 2024-08-31T22:45:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_43 2024-08-31T22:45:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:45:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:45:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:45:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:45:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:45:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_43 2024-08-31T22:45:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:45:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:45:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:45:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:45:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:45:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:45:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:45:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:45:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:45:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:45:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:45:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:29.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:29.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:29.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:45:29.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:45:29.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:29.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:29.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_43 2024-08-31T22:45:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_43 2024-08-31T22:45:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_43 2024-08-31T22:45:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_43 '' 2024-08-31T22:45:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_43 2024-08-31T22:45:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:45:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-08-31T22:45:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_43.status 2024-08-31T22:45:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_43.status 2024-08-31T22:45:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-08-31T22:45:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_43.status 2024-08-31T22:45:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:45:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:45:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7682], [object_number=11, tag_tid=1, entry_tid=7679]]], state=connected] 2024-08-31T22:45:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7682], [object_number=11, tag_tid=1, entry_tid=7679]]], state=connected] 2024-08-31T22:45:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_43.status 2024-08-31T22:45:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_43 2024-08-31T22:45:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_43 2024-08-31T22:45:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_43 '..*' 2024-08-31T22:45:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_43 2024-08-31T22:45:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:45:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-08-31T22:45:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_43.status 2024-08-31T22:45:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_43.status 2024-08-31T22:45:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-08-31T22:45:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_43.status 2024-08-31T22:45:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:45:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:45:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7682], [object_number=11, tag_tid=1, entry_tid=7679]]], state=connected] 2024-08-31T22:45:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7682], [object_number=11, tag_tid=1, entry_tid=7679]]], state=connected] 2024-08-31T22:45:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_43.status 2024-08-31T22:45:29.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:29.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:29.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:29.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-08-31T22:45:29.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:29.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:29.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_43 2024-08-31T22:45:29.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:29.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:29.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:45:29.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-08-31T22:45:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_43': 2024-08-31T22:45:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:45:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:45:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:45:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a87d0c9d2d 2024-08-31T22:45:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a87d0c9d2d 2024-08-31T22:45:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:45:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:45:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:45:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:45:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:39:24 2024 2024-08-31T22:45:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:39:24 2024 2024-08-31T22:45:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:40:25 2024 2024-08-31T22:45:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:45:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a87d0c9d2d 2024-08-31T22:45:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:45:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:45:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 529ea4d1-9425-4590-a421-19ea2e01957e 2024-08-31T22:45:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:45:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_43 snap 2024-08-31T22:45:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_43 2024-08-31T22:45:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:45:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_43.export 2024-08-31T22:45:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_43.export 2024-08-31T22:45:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_43.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_43.export 2024-08-31T22:45:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_43.export 2024-08-31T22:45:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_43.export 2024-08-31T22:45:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_43.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_43.export 2024-08-31T22:45:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_43.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_43.export 2024-08-31T22:45:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:45:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_44 2024-08-31T22:45:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_44 snap 2024-08-31T22:45:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_44 2024-08-31T22:45:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:45:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-08-31T22:45:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_44 2024-08-31T22:45:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-08-31T22:45:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_44 2024-08-31T22:45:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:45:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:35.396 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:45:35.397+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff16b60 2024-08-31T22:45:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-08-31T22:45:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_44 2024-08-31T22:45:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:45:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:45:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:45:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-08-31T22:45:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:45:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:45:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:45:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:45:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:45:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:45:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-08-31T22:45:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:45:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:45:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:45:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:45:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:45:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:45:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:45:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:45:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:45:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:45:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:45:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:45:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:45:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-08-31T22:45:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-08-31T22:45:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:45:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:45:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2024-08-31T22:45:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:45:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:45:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:45:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:45:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:45:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:45:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:45:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:45:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:45:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:45:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:45:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:45:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:45:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:45:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:45:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:45:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:45:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:45:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:45:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:45:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:45:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:45:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2024-08-31T22:45:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:45:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:45:35.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:45:35.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:45:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:45:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:45:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:45:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2024-08-31T22:45:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_44 2024-08-31T22:45:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:45:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_44 2024-08-31T22:45:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_44 2024-08-31T22:45:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:45:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:45:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:45:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:45:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:45:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_44 2024-08-31T22:45:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_44 2024-08-31T22:45:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:45:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_44 ']' 2024-08-31T22:45:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_44' 2024-08-31T22:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2024-08-31T22:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:45:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2024-08-31T22:45:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:45:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:45:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:45:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:45:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:45:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:45:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:45:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:45:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:45:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:45:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2024-08-31T22:45:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-08-31T22:45:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:45:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:45:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_44 2024-08-31T22:45:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:45:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:45:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:45:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:45:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:45:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:45:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:45:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:45:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:45:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:45:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:45:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:45:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:45:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:45:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:45:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:45:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:45:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:45:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:45:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:45:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:45:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:45:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:36.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:36.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:36.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_44 2024-08-31T22:45:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:36.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_44 2024-08-31T22:45:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_44 2024-08-31T22:45:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_44 '' 2024-08-31T22:45:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_44 2024-08-31T22:45:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:45:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-08-31T22:45:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_44.status 2024-08-31T22:45:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_44.status 2024-08-31T22:45:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-08-31T22:45:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_44.status 2024-08-31T22:45:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:45:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:45:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9007], [object_number=10, tag_tid=1, entry_tid=9006]]], state=connected] 2024-08-31T22:45:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9007], [object_number=10, tag_tid=1, entry_tid=9006]]], state=connected] 2024-08-31T22:45:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_44.status 2024-08-31T22:45:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2024-08-31T22:45:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_44 2024-08-31T22:45:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2024-08-31T22:45:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_44 2024-08-31T22:45:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:45:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-08-31T22:45:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_44.status 2024-08-31T22:45:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_44.status 2024-08-31T22:45:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-08-31T22:45:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_44.status 2024-08-31T22:45:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:45:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:45:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9007], [object_number=10, tag_tid=1, entry_tid=9006]]], state=connected] 2024-08-31T22:45:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9007], [object_number=10, tag_tid=1, entry_tid=9006]]], state=connected] 2024-08-31T22:45:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_44.status 2024-08-31T22:45:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-08-31T22:45:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_44 2024-08-31T22:45:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:45:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-08-31T22:45:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_44': 2024-08-31T22:45:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:45:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:45:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:45:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10ab9fbb03b4 2024-08-31T22:45:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10ab9fbb03b4 2024-08-31T22:45:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:45:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:45:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:45:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:45:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:39:34 2024 2024-08-31T22:45:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:39:34 2024 2024-08-31T22:45:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:40:34 2024 2024-08-31T22:45:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:45:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10ab9fbb03b4 2024-08-31T22:45:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:45:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:45:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: b193b278-5845-41e3-8e62-cc111da56c58 2024-08-31T22:45:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:45:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_44 snap 2024-08-31T22:45:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_44 2024-08-31T22:45:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:45:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_44.export 2024-08-31T22:45:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_44.export 2024-08-31T22:45:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_44.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_44.export 2024-08-31T22:45:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_44.export 2024-08-31T22:45:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_44.export 2024-08-31T22:45:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_44.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_44.export 2024-08-31T22:45:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_44.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_44.export 2024-08-31T22:45:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:45:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_45 2024-08-31T22:45:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_45 snap 2024-08-31T22:45:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_45 2024-08-31T22:45:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:45:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-08-31T22:45:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_45 2024-08-31T22:45:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-08-31T22:45:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_45 2024-08-31T22:45:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:45:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-08-31T22:45:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_45 2024-08-31T22:45:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:45:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:45:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:45:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-08-31T22:45:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:45:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:45:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:45:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:45:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:45:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:45:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-08-31T22:45:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:45:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:45:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:45:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:45:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:45:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:45:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:45:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:45:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:45:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:45:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:45:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:45:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:45:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-08-31T22:45:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:45:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:45:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:45:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:45:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:45:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:45:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:45:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2024-08-31T22:45:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_45 2024-08-31T22:45:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:45:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_45 2024-08-31T22:45:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_45 2024-08-31T22:45:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:45:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:45:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:45:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:45:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:45:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_45 2024-08-31T22:45:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_45 2024-08-31T22:45:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:45:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_45 ']' 2024-08-31T22:45:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_45' 2024-08-31T22:45:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2024-08-31T22:45:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:45:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:45:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:45:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:45:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:45:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2024-08-31T22:45:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:45:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:45:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:45:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:45:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:45:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:45:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:45:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:45:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:45:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:45:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:45:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:45:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:45:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2024-08-31T22:45:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_45 2024-08-31T22:45:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_45 2024-08-31T22:45:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_45 '' 2024-08-31T22:45:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_45 2024-08-31T22:45:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:45:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-08-31T22:45:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_45.status 2024-08-31T22:45:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:41.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_45.status 2024-08-31T22:45:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_45.status 2024-08-31T22:45:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-08-31T22:45:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:45:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:45:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6062], [object_number=7, tag_tid=1, entry_tid=6059]]], state=connected] 2024-08-31T22:45:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6062], [object_number=7, tag_tid=1, entry_tid=6059]]], state=connected] 2024-08-31T22:45:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_45.status 2024-08-31T22:45:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:45:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2024-08-31T22:45:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_45 2024-08-31T22:45:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2024-08-31T22:45:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_45 2024-08-31T22:45:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:45:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-08-31T22:45:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_45.status 2024-08-31T22:45:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_45.status 2024-08-31T22:45:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-08-31T22:45:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_45.status 2024-08-31T22:45:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:45:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:45:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6062], [object_number=7, tag_tid=1, entry_tid=6059]]], state=connected] 2024-08-31T22:45:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6062], [object_number=7, tag_tid=1, entry_tid=6059]]], state=connected] 2024-08-31T22:45:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_45.status 2024-08-31T22:45:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:45:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-08-31T22:45:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-08-31T22:45:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_45 2024-08-31T22:45:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:45:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-08-31T22:45:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_45': 2024-08-31T22:45:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:45:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:45:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:45:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a871bc5a86 2024-08-31T22:45:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a871bc5a86 2024-08-31T22:45:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:45:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:45:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:45:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:45:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:39:46 2024 2024-08-31T22:45:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:39:46 2024 2024-08-31T22:45:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:39:46 2024 2024-08-31T22:45:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:45:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a871bc5a86 2024-08-31T22:45:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:45:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:45:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: d34c9c26-536a-4911-810a-56df140f507e 2024-08-31T22:45:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:45:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_45 snap 2024-08-31T22:45:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_45 2024-08-31T22:45:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:45:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_45.export 2024-08-31T22:45:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_45.export 2024-08-31T22:45:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_45.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_45.export 2024-08-31T22:45:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_45.export 2024-08-31T22:45:42.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_45.export 2024-08-31T22:45:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_45.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_45.export 2024-08-31T22:45:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_45.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_45.export 2024-08-31T22:45:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:45:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_46 2024-08-31T22:45:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_46 snap 2024-08-31T22:45:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_46 2024-08-31T22:45:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:45:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-08-31T22:45:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_46 2024-08-31T22:45:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-08-31T22:45:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_46 2024-08-31T22:45:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:45:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-08-31T22:45:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_46 2024-08-31T22:45:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:45:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:45:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:45:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-08-31T22:45:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:45:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:45:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:45:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:45:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:45:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:45:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-08-31T22:45:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:45:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:45:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:45:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:45:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:45:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:45:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:45:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-08-31T22:45:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-08-31T22:45:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:45:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:45:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-08-31T22:45:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:45:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:45:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:45:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:45:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:45:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:45:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:45:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:45:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:45:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:45:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:45:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:45:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:45:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:45:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:45:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:45:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:45:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:45:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:45:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:45:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:45:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:45:46.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-08-31T22:45:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:45:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:45:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:45:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:45:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:45:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:45:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:45:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2024-08-31T22:45:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_46 2024-08-31T22:45:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:45:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_46 2024-08-31T22:45:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_46 2024-08-31T22:45:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:45:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:45:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:45:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:45:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:45:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_46 2024-08-31T22:45:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_46 2024-08-31T22:45:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:45:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_46 ']' 2024-08-31T22:45:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_46' 2024-08-31T22:45:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2024-08-31T22:45:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:45:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:45:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:45:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:45:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:45:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2024-08-31T22:45:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:45:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:45:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:45:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:45:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:45:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:45:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:45:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:45:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:45:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:45:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:45:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:45:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:45:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2024-08-31T22:45:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-08-31T22:45:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:45:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:45:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2024-08-31T22:45:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:45:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:45:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:45:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:45:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:45:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:45:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:45:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:45:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:46.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:45:46.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:46.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:45:46.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:45:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:45:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:45:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:45:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:45:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:45:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:45:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:45:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:45:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:45:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:45:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:45:46.793 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2024-08-31T22:45:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_46 2024-08-31T22:45:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_46 2024-08-31T22:45:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_46 '' 2024-08-31T22:45:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_46 2024-08-31T22:45:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:45:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-08-31T22:45:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_46.status 2024-08-31T22:45:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_46.status 2024-08-31T22:45:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-08-31T22:45:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_46.status 2024-08-31T22:45:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:45:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:45:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8467], [object_number=10, tag_tid=1, entry_tid=8466]]], state=connected] 2024-08-31T22:45:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8467], [object_number=10, tag_tid=1, entry_tid=8466]]], state=connected] 2024-08-31T22:45:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_46.status 2024-08-31T22:45:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2024-08-31T22:45:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_46 2024-08-31T22:45:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2024-08-31T22:45:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_46 2024-08-31T22:45:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:45:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-08-31T22:45:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_46.status 2024-08-31T22:45:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_46.status 2024-08-31T22:45:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-08-31T22:45:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_46.status 2024-08-31T22:45:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 2 2024-08-31T22:45:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 2 2024-08-31T22:45:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8467], [object_number=10, tag_tid=1, entry_tid=8466]]], state=connected] 2024-08-31T22:45:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8467], [object_number=10, tag_tid=1, entry_tid=8466]]], state=connected] 2024-08-31T22:45:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_46.status 2024-08-31T22:45:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-08-31T22:45:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-08-31T22:45:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_46 2024-08-31T22:45:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:45:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-08-31T22:45:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_46': 2024-08-31T22:45:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:45:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:45:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:45:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10abcf9cbf7b 2024-08-31T22:45:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10abcf9cbf7b 2024-08-31T22:45:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:45:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:45:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:45:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:45:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:39:54 2024 2024-08-31T22:45:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:39:54 2024 2024-08-31T22:45:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:40:54 2024 2024-08-31T22:45:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:45:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10abcf9cbf7b 2024-08-31T22:45:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:45:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:45:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 735aa208-4a53-4fe9-8f00-a04a15aca96e 2024-08-31T22:45:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:45:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_46 snap 2024-08-31T22:45:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_46 2024-08-31T22:45:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:45:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_46.export 2024-08-31T22:45:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_46.export 2024-08-31T22:45:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_46.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_46.export 2024-08-31T22:45:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_46.export 2024-08-31T22:45:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_46.export 2024-08-31T22:45:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_46.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_46.export 2024-08-31T22:45:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_46.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_46.export 2024-08-31T22:45:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:45:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_47 2024-08-31T22:45:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_47 snap 2024-08-31T22:45:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_47 2024-08-31T22:45:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:45:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-08-31T22:45:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_47 2024-08-31T22:45:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-08-31T22:45:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_47 2024-08-31T22:45:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:45:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-08-31T22:45:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_47 2024-08-31T22:45:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:45:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:45:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:45:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-08-31T22:45:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:45:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:45:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:45:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:45:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:45:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:45:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-08-31T22:45:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:45:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:45:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:45:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:45:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:45:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:45:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:45:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:45:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:45:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:45:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:45:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:45:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:45:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-08-31T22:45:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:45:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:45:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:45:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:45:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:45:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:45:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:45:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2024-08-31T22:45:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_47 2024-08-31T22:45:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:45:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_47 2024-08-31T22:45:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_47 2024-08-31T22:45:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:45:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:45:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:45:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:45:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:45:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_47 2024-08-31T22:45:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_47 2024-08-31T22:45:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:45:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_47 ']' 2024-08-31T22:45:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_47' 2024-08-31T22:45:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_47 2024-08-31T22:45:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:45:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:45:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:45:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:45:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:45:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_47 2024-08-31T22:45:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:45:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:45:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:45:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:45:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:45:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:45:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:45:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:45:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:45:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:45:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:45:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:45:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:45:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_47 2024-08-31T22:45:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_47 2024-08-31T22:45:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_47 2024-08-31T22:45:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_47 '' 2024-08-31T22:45:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_47 2024-08-31T22:45:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:45:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-08-31T22:45:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_47.status 2024-08-31T22:45:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_47.status 2024-08-31T22:45:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-08-31T22:45:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_47.status 2024-08-31T22:45:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 0 2024-08-31T22:45:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:45:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3471], [object_number=2, tag_tid=1, entry_tid=3470]]], state=connected] 2024-08-31T22:45:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=0, tag_tid=1, entry_tid=3472], [object_number=3, tag_tid=1, entry_tid=3471], [object_number=2, tag_tid=1, entry_tid=3470], [object_number=1, tag_tid=1, entry_tid=3469]]], state=connected] 2024-08-31T22:45:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_47.status 2024-08-31T22:45:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:45:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_47 2024-08-31T22:45:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_47 2024-08-31T22:45:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_47 '..*' 2024-08-31T22:45:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_47 2024-08-31T22:45:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:45:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-08-31T22:45:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_47.status 2024-08-31T22:45:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_47.status 2024-08-31T22:45:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-08-31T22:45:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_47.status 2024-08-31T22:45:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 0 2024-08-31T22:45:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:45:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3471], [object_number=2, tag_tid=1, entry_tid=3470]]], state=connected] 2024-08-31T22:45:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=0, tag_tid=1, entry_tid=3472], [object_number=3, tag_tid=1, entry_tid=3471], [object_number=2, tag_tid=1, entry_tid=3470], [object_number=1, tag_tid=1, entry_tid=3469]]], state=connected] 2024-08-31T22:45:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_47.status 2024-08-31T22:45:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=0, tag_tid=1, entry_tid=3472' 2024-08-31T22:45:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=0, tag_tid=1, entry_tid=3472' 2024-08-31T22:45:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=0, tag_tid=1, entry_tid=3472' '!=' '' 2024-08-31T22:45:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:45:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=0, tag_tid=1, entry_tid=3472' = '' 2024-08-31T22:45:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=0, tag_tid=1, entry_tid=3472' 2024-08-31T22:45:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:45:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:45:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:45:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=3 2024-08-31T22:45:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=0, tag_tid=1, entry_tid=3472' 2024-08-31T22:45:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:45:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:45:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=1 2024-08-31T22:45:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:45:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:45:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:45:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:45:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=0, tag_tid=1, entry_tid=3472' 2024-08-31T22:45:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:45:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:45:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=3472 2024-08-31T22:45:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 3 = 1 -a 1 -le 3472 2024-08-31T22:45:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:45:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:45:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:45:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_47 2024-08-31T22:45:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_47 2024-08-31T22:45:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:45:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_47 ']' 2024-08-31T22:45:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_47' 2024-08-31T22:45:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_47 2024-08-31T22:45:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:45:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:45:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:45:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:45:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:45:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_47 2024-08-31T22:45:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:45:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:45:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:45:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:45:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:45:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:45:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:45:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:45:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:45:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:45:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:45:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:45:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:45:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_47 2024-08-31T22:45:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_47 2024-08-31T22:45:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_47 2024-08-31T22:45:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_47 '' 2024-08-31T22:45:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_47 2024-08-31T22:45:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:45:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-08-31T22:45:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_47.status 2024-08-31T22:45:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_47.status 2024-08-31T22:45:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-08-31T22:45:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_47.status 2024-08-31T22:45:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 0 2024-08-31T22:45:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:45:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3471], [object_number=2, tag_tid=1, entry_tid=3470]]], state=connected] 2024-08-31T22:45:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3471], [object_number=2, tag_tid=1, entry_tid=3470]]], state=connected] 2024-08-31T22:45:52.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_47.status 2024-08-31T22:45:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:45:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_47 2024-08-31T22:45:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_47 2024-08-31T22:45:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_47 '..*' 2024-08-31T22:45:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_47 2024-08-31T22:45:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:45:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-08-31T22:45:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_47.status 2024-08-31T22:45:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:52.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_47.status 2024-08-31T22:45:52.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-08-31T22:45:52.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_47.status 2024-08-31T22:45:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 0 2024-08-31T22:45:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:45:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3471], [object_number=2, tag_tid=1, entry_tid=3470]]], state=connected] 2024-08-31T22:45:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3471], [object_number=2, tag_tid=1, entry_tid=3470]]], state=connected] 2024-08-31T22:45:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_47.status 2024-08-31T22:45:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:45:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:45:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-08-31T22:45:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_47 2024-08-31T22:45:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:45:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-08-31T22:45:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_47': 2024-08-31T22:45:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:45:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:45:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:45:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a8866ec92a 2024-08-31T22:45:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a8866ec92a 2024-08-31T22:45:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:45:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:45:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:45:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:45:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:40:03 2024 2024-08-31T22:45:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:40:03 2024 2024-08-31T22:45:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:40:03 2024 2024-08-31T22:45:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:45:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a8866ec92a 2024-08-31T22:45:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:45:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:45:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: bace054c-fc80-498d-acfd-1f1908919254 2024-08-31T22:45:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:45:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_47 snap 2024-08-31T22:45:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_47 2024-08-31T22:45:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:45:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_47.export 2024-08-31T22:45:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_47.export 2024-08-31T22:45:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_47.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_47.export 2024-08-31T22:45:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_47.export 2024-08-31T22:45:53.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_47.export 2024-08-31T22:45:53.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_47.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_47.export 2024-08-31T22:45:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_47.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_47.export 2024-08-31T22:45:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:45:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_48 2024-08-31T22:45:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_48 snap 2024-08-31T22:45:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_48 2024-08-31T22:45:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:45:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-08-31T22:45:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_48 2024-08-31T22:45:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-08-31T22:45:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_48 2024-08-31T22:45:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:45:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-08-31T22:45:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_48 2024-08-31T22:45:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:45:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:45:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:45:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-08-31T22:45:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:45:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:45:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:45:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:45:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:45:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:45:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-08-31T22:45:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:45:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:45:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:45:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:45:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:45:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:45:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:45:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:45:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:45:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:45:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-08-31T22:45:56.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-08-31T22:45:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:45:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:45:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2024-08-31T22:45:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:45:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:45:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:45:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:45:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:45:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:45:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:45:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:45:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:45:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:45:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:45:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:45:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:45:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:45:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:45:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:45:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:45:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:45:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:45:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:45:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:45:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:45:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:45:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:45:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2024-08-31T22:45:56.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:45:56.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:45:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:45:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:45:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:45:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:45:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:45:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2024-08-31T22:45:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_48 2024-08-31T22:45:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:45:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_48 2024-08-31T22:45:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:45:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_48 2024-08-31T22:45:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:45:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:45:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:45:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:45:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:45:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_48 2024-08-31T22:45:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_48 2024-08-31T22:45:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:45:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_48 ']' 2024-08-31T22:45:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-08-31T22:45:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-08-31T22:45:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:45:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:45:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:45:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:45:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:45:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:45:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:45:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-08-31T22:45:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:45:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:45:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:45:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:45:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:45:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:45:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:45:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:45:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:45:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:45:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:45:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:45:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:45:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:45:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:45:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:45:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:45:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:45:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:45:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:45:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:45:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:45:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:56.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:56.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:45:56.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-08-31T22:45:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-08-31T22:45:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:45:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:45:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:45:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-08-31T22:45:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:45:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:45:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:45:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:45:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:45:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:45:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:45:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:45:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:45:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:45:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:45:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:45:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:45:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:45:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:45:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:45:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:45:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:45:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:45:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:45:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:45:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:45:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:45:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:45:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:45:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:45:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:45:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:45:56.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:45:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-08-31T22:45:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:57.097 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-08-31T22:45:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_48 2024-08-31T22:45:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-08-31T22:45:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_48 2024-08-31T22:45:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:45:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-08-31T22:45:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_48.status 2024-08-31T22:45:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_48.status 2024-08-31T22:45:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-08-31T22:45:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_48.status 2024-08-31T22:45:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:45:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:45:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4234], [object_number=7, tag_tid=1, entry_tid=4231]]], state=connected] 2024-08-31T22:45:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4234], [object_number=7, tag_tid=1, entry_tid=4231]]], state=connected] 2024-08-31T22:45:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_48.status 2024-08-31T22:45:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:45:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-08-31T22:45:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_48 2024-08-31T22:45:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-08-31T22:45:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:45:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:45:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_48 2024-08-31T22:45:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:45:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-08-31T22:45:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_48.status 2024-08-31T22:45:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:45:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_48.status 2024-08-31T22:45:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-08-31T22:45:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_48.status 2024-08-31T22:45:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:45:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:45:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:45:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4234], [object_number=7, tag_tid=1, entry_tid=4231]]], state=connected] 2024-08-31T22:45:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4234], [object_number=7, tag_tid=1, entry_tid=4231]]], state=connected] 2024-08-31T22:45:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_48.status 2024-08-31T22:45:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:45:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:45:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-08-31T22:45:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:45:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_48 2024-08-31T22:45:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:45:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:45:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:45:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-08-31T22:45:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_48': 2024-08-31T22:45:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:45:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:45:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:45:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10ab3059fe08 2024-08-31T22:45:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10ab3059fe08 2024-08-31T22:45:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:45:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:45:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:45:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:45:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:40:09 2024 2024-08-31T22:45:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:40:09 2024 2024-08-31T22:45:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:40:09 2024 2024-08-31T22:45:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:45:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10ab3059fe08 2024-08-31T22:45:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:45:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:45:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 41798d42-6ff9-4e33-bfd1-cd5b1ffee017 2024-08-31T22:45:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:45:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_48 snap 2024-08-31T22:45:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_48 2024-08-31T22:45:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:45:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_48.export 2024-08-31T22:45:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_48.export 2024-08-31T22:45:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_48.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_48.export 2024-08-31T22:45:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_48.export 2024-08-31T22:45:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_48.export 2024-08-31T22:45:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:45:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_48.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_48.export 2024-08-31T22:45:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_48.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_48.export 2024-08-31T22:45:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:45:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:45:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_49 2024-08-31T22:45:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_49 snap 2024-08-31T22:45:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:45:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:45:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_49 2024-08-31T22:45:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:45:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:46:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-08-31T22:46:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:46:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_49 2024-08-31T22:46:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-08-31T22:46:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:46:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_49 2024-08-31T22:46:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:46:01.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:46:01.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:46:01.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:46:02.233 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-08-31T22:46:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:46:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_49 2024-08-31T22:46:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:46:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:46:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:46:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-08-31T22:46:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:46:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:46:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:46:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:46:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:46:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:46:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:46:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:46:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:46:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:46:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:46:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-08-31T22:46:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:46:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:46:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:46:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:46:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:46:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:46:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:46:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:46:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:46:02.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:46:02.240 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:46:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:46:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:46:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:46:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:46:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:46:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:46:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:46:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:46:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:46:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:46:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:46:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:46:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:46:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:46:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:46:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:46:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:46:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:46:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:46:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:46:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:46:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:46:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-08-31T22:46:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:46:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:46:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:46:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:46:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:46:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:46:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:46:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2024-08-31T22:46:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:46:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_49 2024-08-31T22:46:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:46:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_49 2024-08-31T22:46:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:46:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_49 2024-08-31T22:46:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:46:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:46:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:46:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:46:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:46:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_49 2024-08-31T22:46:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:46:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_49 2024-08-31T22:46:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:46:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_49 ']' 2024-08-31T22:46:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_49' 2024-08-31T22:46:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2024-08-31T22:46:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:46:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:46:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:46:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:46:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:46:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:46:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:46:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:46:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:46:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2024-08-31T22:46:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:46:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:46:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:46:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:46:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:46:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:46:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:46:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:46:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:46:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:46:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:46:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:46:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:46:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:46:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:46:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:46:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:46:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:46:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:46:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:46:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:46:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:46:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:46:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:46:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:46:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:46:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:46:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:46:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:46:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:46:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:46:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:46:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:46:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2024-08-31T22:46:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_49 2024-08-31T22:46:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:46:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:46:02.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_49 2024-08-31T22:46:02.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_49 '' 2024-08-31T22:46:02.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:46:02.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:46:02.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_49 2024-08-31T22:46:02.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:46:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-08-31T22:46:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_49.status 2024-08-31T22:46:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:46:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_49.status 2024-08-31T22:46:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-08-31T22:46:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_49.status 2024-08-31T22:46:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:46:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:46:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:46:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11423], [object_number=14, tag_tid=1, entry_tid=11422]]], state=connected] 2024-08-31T22:46:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=11424], [object_number=15, tag_tid=1, entry_tid=11423], [object_number=14, tag_tid=1, entry_tid=11422], [object_number=13, tag_tid=1, entry_tid=11421]]], state=connected] 2024-08-31T22:46:02.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_49.status 2024-08-31T22:46:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:46:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2024-08-31T22:46:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:46:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:46:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_49 2024-08-31T22:46:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2024-08-31T22:46:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:46:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:46:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_49 2024-08-31T22:46:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:46:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-08-31T22:46:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_49.status 2024-08-31T22:46:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:46:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_49.status 2024-08-31T22:46:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-08-31T22:46:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_49.status 2024-08-31T22:46:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:46:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:46:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:46:02.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11423], [object_number=14, tag_tid=1, entry_tid=11422]]], state=connected] 2024-08-31T22:46:02.896 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=11424], [object_number=15, tag_tid=1, entry_tid=11423], [object_number=14, tag_tid=1, entry_tid=11422], [object_number=13, tag_tid=1, entry_tid=11421]]], state=connected] 2024-08-31T22:46:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_49.status 2024-08-31T22:46:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=12, tag_tid=1, entry_tid=11424' 2024-08-31T22:46:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=12, tag_tid=1, entry_tid=11424' 2024-08-31T22:46:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=12, tag_tid=1, entry_tid=11424' '!=' '' 2024-08-31T22:46:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:46:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 'object_number=12, tag_tid=1, entry_tid=11424' = '' 2024-08-31T22:46:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ last_mirror_pos='object_number=12, tag_tid=1, entry_tid=11424' 2024-08-31T22:46:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:46:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:46:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:46:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_tag=2 2024-08-31T22:46:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=12, tag_tid=1, entry_tid=11424' 2024-08-31T22:46:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-08-31T22:46:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:46:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_tag=1 2024-08-31T22:46:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:46:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:46:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:46:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_entry=1 2024-08-31T22:46:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo 'object_number=12, tag_tid=1, entry_tid=11424' 2024-08-31T22:46:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-08-31T22:46:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut -d= -f 2 2024-08-31T22:46:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_entry=11424 2024-08-31T22:46:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 2 = 1 -a 1 -le 11424 2024-08-31T22:46:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:46:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:46:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:46:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_49 2024-08-31T22:46:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:46:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_49 2024-08-31T22:46:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:46:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_49 ']' 2024-08-31T22:46:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_49' 2024-08-31T22:46:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2024-08-31T22:46:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:46:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:46:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:46:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:46:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:46:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:46:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:46:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:46:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:46:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2024-08-31T22:46:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:46:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:46:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:46:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:46:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:46:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:46:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:46:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:46:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:46:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:46:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:46:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:46:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:46:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:46:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:46:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:46:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:46:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:46:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:46:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:46:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:46:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:46:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:46:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:46:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:46:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:46:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:46:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:46:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:46:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:46:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:46:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:46:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:46:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2024-08-31T22:46:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_49 2024-08-31T22:46:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:46:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:46:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_49 2024-08-31T22:46:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_49 '' 2024-08-31T22:46:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:46:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:46:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_49 2024-08-31T22:46:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:46:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-08-31T22:46:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_49.status 2024-08-31T22:46:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:46:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_49.status 2024-08-31T22:46:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-08-31T22:46:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_49.status 2024-08-31T22:46:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:46:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:46:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:46:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11423], [object_number=14, tag_tid=1, entry_tid=11422]]], state=connected] 2024-08-31T22:46:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11423], [object_number=14, tag_tid=1, entry_tid=11422]]], state=connected] 2024-08-31T22:46:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_49.status 2024-08-31T22:46:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:46:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2024-08-31T22:46:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:46:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:46:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_49 2024-08-31T22:46:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2024-08-31T22:46:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:46:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:46:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_49 2024-08-31T22:46:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:46:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-08-31T22:46:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_49.status 2024-08-31T22:46:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:46:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_49.status 2024-08-31T22:46:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-08-31T22:46:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_49.status 2024-08-31T22:46:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 3 2024-08-31T22:46:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 3 2024-08-31T22:46:03.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:46:03.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11423], [object_number=14, tag_tid=1, entry_tid=11422]]], state=connected] 2024-08-31T22:46:03.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11423], [object_number=14, tag_tid=1, entry_tid=11422]]], state=connected] 2024-08-31T22:46:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_49.status 2024-08-31T22:46:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:46:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-08-31T22:46:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-08-31T22:46:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:46:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_49 2024-08-31T22:46:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:46:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:46:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:46:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:46:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-08-31T22:46:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_49': 2024-08-31T22:46:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:46:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:46:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:46:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10a8a9ac7f94 2024-08-31T22:46:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10a8a9ac7f94 2024-08-31T22:46:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:46:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:46:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:46:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:46:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:40:17 2024 2024-08-31T22:46:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:40:17 2024 2024-08-31T22:46:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:41:17 2024 2024-08-31T22:46:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:46:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10a8a9ac7f94 2024-08-31T22:46:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:46:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:46:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 79a0cda3-d042-4100-8306-23459a6d8d56 2024-08-31T22:46:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:46:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_49 snap 2024-08-31T22:46:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_49 2024-08-31T22:46:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:46:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:46:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_49.export 2024-08-31T22:46:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_49.export 2024-08-31T22:46:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_49.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_49.export 2024-08-31T22:46:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_49.export 2024-08-31T22:46:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:46:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_49.export 2024-08-31T22:46:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:46:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_49.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_49.export 2024-08-31T22:46:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_49.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_49.export 2024-08-31T22:46:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_50 2024-08-31T22:46:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_snap cluster2 mirror image_50 snap 2024-08-31T22:46:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_50 2024-08-31T22:46:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:46:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-08-31T22:46:06.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-08-31T22:46:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:46:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_50 2024-08-31T22:46:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-08-31T22:46:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:46:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_50 2024-08-31T22:46:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=started 2024-08-31T22:46:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:46:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:46:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:46:07.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-08-31T22:46:07.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:46:07.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:07.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_50 2024-08-31T22:46:07.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=started 2024-08-31T22:46:07.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local status_result 2024-08-31T22:46:07.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=stopped 2024-08-31T22:46:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-08-31T22:46:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_instance=cluster1 2024-08-31T22:46:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:46:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -i state 2024-08-31T22:46:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:46:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance=cluster1 2024-08-31T22:46:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local loop_instance 2024-08-31T22:46:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:46:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sleep 0 2024-08-31T22:46:07.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:46:07.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ seq 0 1 2024-08-31T22:46:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:46:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-08-31T22:46:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:46:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:46:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:46:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:46:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:46:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:46:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:46:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:46:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:46:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:46:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:46:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:46:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:46:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:46:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:46:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:0 2024-08-31T22:46:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:46:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:46:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:46:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:46:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:46:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:46:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:46:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:46:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:46:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:46:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:46:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:46:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:46:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:46:07.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:46:07.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:46:07.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:46:07.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-08-31T22:46:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-08-31T22:46:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror status : get status for rbd mirror 2024-08-31T22:46:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:46:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:46:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-08-31T22:46:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:46:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ shift 2024-08-31T22:46:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:46:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:46:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:46:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:46:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:46:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:46:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:46:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:46:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:46:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:46:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:46:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:46:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:46:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local local_cluster=cluster1:1 2024-08-31T22:46:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1 2024-08-31T22:46:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:46:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:46:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:46:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=local_cluster 2024-08-31T22:46:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:46:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:46:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:46:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:46:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval local_cluster=cluster1 2024-08-31T22:46:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ local_cluster=cluster1 2024-08-31T22:46:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:46:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:46:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:46:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:46:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:46:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:46:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-08-31T22:46:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ return 0 2024-08-31T22:46:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ status_result=' "state": "Replaying"' 2024-08-31T22:46:07.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo ' "state": "Replaying"' 2024-08-31T22:46:07.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ grep -i Replaying 2024-08-31T22:46:07.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: "state": "Replaying" 2024-08-31T22:46:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=started 2024-08-31T22:46:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test started = started 2024-08-31T22:46:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2024-08-31T22:46:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:46:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_50 2024-08-31T22:46:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' journal = journal ']' 2024-08-31T22:46:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_50 2024-08-31T22:46:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local local_cluster=cluster1 2024-08-31T22:46:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_50 2024-08-31T22:46:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-08-31T22:46:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-08-31T22:46:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:46:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-08-31T22:46:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.2 2024-08-31T22:46:07.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ flush cluster1 mirror image_50 2024-08-31T22:46:07.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:46:07.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:07.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_50 2024-08-31T22:46:07.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local 'cmd=rbd mirror flush' 2024-08-31T22:46:07.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n image_50 ']' 2024-08-31T22:46:07.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmd='rbd mirror flush mirror/image_50' 2024-08-31T22:46:07.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_50 2024-08-31T22:46:07.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_instance=cluster1 2024-08-31T22:46:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:46:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:46:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance=cluster1 2024-08-31T22:46:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loop_instance 2024-08-31T22:46:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-08-31T22:46:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-08-31T22:46:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:46:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:46:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_50 2024-08-31T22:46:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:46:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:46:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:46:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:46:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:0 2024-08-31T22:46:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:46:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:46:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:46:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:46:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 = cluster1:0 ']' 2024-08-31T22:46:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:46:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:46:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:46:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:46:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:46:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:46:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:46:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:46:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:46:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:46:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:46:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:46:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:46:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:46:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:46:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:46:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:46:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:46:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:46:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:46:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:46:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:46:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:46:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_50 2024-08-31T22:46:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-08-31T22:46:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd mirror flush : flush rbd mirror 2024-08-31T22:46:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:admin_socket: invalid command 2024-08-31T22:46:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:46:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_50 2024-08-31T22:46:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:46:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:46:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:46:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:46:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1:1 2024-08-31T22:46:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:46:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:46:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:46:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=1 2024-08-31T22:46:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 1 = cluster1:1 ']' 2024-08-31T22:46:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:46:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:46:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=1 2024-08-31T22:46:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:46:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:46:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:46:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:46:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:46:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:46:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:46:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:46:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:46:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:46:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:46:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:46:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:46:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:46:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:46:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:46:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:46:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:46:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:46:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:46:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_50 2024-08-31T22:46:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_master_journal_position cluster2 mirror image_50 2024-08-31T22:46:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:46:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:46:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_50 2024-08-31T22:46:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_50 '' 2024-08-31T22:46:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:46:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:46:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_50 2024-08-31T22:46:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local id_regexp= 2024-08-31T22:46:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-08-31T22:46:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_50.status 2024-08-31T22:46:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:46:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_50.status 2024-08-31T22:46:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-08-31T22:46:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_50.status 2024-08-31T22:46:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:46:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:46:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:46:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4971], [object_number=6, tag_tid=1, entry_tid=4970]]], state=connected] 2024-08-31T22:46:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4971], [object_number=6, tag_tid=1, entry_tid=4970]]], state=connected] 2024-08-31T22:46:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_50.status 2024-08-31T22:46:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:46:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_mirror_journal_position cluster2 mirror image_50 2024-08-31T22:46:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:46:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:46:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_50 2024-08-31T22:46:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_journal_position cluster2 mirror image_50 '..*' 2024-08-31T22:46:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:46:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:46:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_50 2024-08-31T22:46:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local 'id_regexp=..*' 2024-08-31T22:46:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-08-31T22:46:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo cluster2-mirror-image_50.status 2024-08-31T22:46:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ sed -e 's|[/ ]|_|g' 2024-08-31T22:46:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local status_log=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_50.status 2024-08-31T22:46:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-08-31T22:46:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ tee /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_50.status 2024-08-31T22:46:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:minimum_set: 1 2024-08-31T22:46:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:active_set: 1 2024-08-31T22:46:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:registered clients: 2024-08-31T22:46:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4971], [object_number=6, tag_tid=1, entry_tid=4970]]], state=connected] 2024-08-31T22:46:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: [id=ad196c8b-4b97-4cf9-85e0-cd944c1b883d, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4971], [object_number=6, tag_tid=1, entry_tid=4970]]], state=connected] 2024-08-31T22:46:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_50.status 2024-08-31T22:46:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:46:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-08-31T22:46:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-08-31T22:46:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:46:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_50 2024-08-31T22:46:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:46:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:46:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-08-31T22:46:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:46:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-08-31T22:46:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_50': 2024-08-31T22:46:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 128 MiB in 32 objects 2024-08-31T22:46:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:46:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 1 2024-08-31T22:46:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 10abf73a06ad 2024-08-31T22:46:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.10abf73a06ad 2024-08-31T22:46:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:46:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:46:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:46:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:46:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:40:30 2024 2024-08-31T22:46:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:40:30 2024 2024-08-31T22:46:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:40:30 2024 2024-08-31T22:46:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: protected: False 2024-08-31T22:46:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 10abf73a06ad 2024-08-31T22:46:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:46:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:46:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 7facd974-5ac3-4cd4-824c-a61813c604aa 2024-08-31T22:46:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: false 2024-08-31T22:46:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ compare_image_snaps mirror image_50 snap 2024-08-31T22:46:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_50 2024-08-31T22:46:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap_name=snap 2024-08-31T22:46:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local ret=0 2024-08-31T22:46:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local rmt_export=/tmp/tmp.iYfzsjLopX/cluster2-mirror-image_50.export 2024-08-31T22:46:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local loc_export=/tmp/tmp.iYfzsjLopX/cluster1-mirror-image_50.export 2024-08-31T22:46:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_50.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_50.export 2024-08-31T22:46:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_50.export 2024-08-31T22:46:09.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:46:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_50.export 2024-08-31T22:46:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-08-31T22:46:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cmp /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_50.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_50.export 2024-08-31T22:46:09.970 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /tmp/tmp.iYfzsjLopX/cluster2-mirror-image_50.export /tmp/tmp.iYfzsjLopX/cluster1-mirror-image_50.export 2024-08-31T22:46:10.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:10.012 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ testlog 'TEST: delete many images' 2024-08-31T22:46:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ tee -a /tmp/tmp.iYfzsjLopX/rbd-mirror.test.log 2024-08-31T22:46:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ date '+%F %T' 2024-08-31T22:46:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo 2024-08-31 22:46:10 TEST: delete many images 2024-08-31T22:46:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:2024-08-31 22:46:10 TEST: delete many images 2024-08-31T22:46:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 1 50 2024-08-31T22:46:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_1 2024-08-31T22:46:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-08-31T22:46:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:46:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-08-31T22:46:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_1 2024-08-31T22:46:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:46:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_1 2024-08-31T22:46:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:46:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-08-31T22:46:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-08-31T22:46:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_2 2024-08-31T22:46:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-08-31T22:46:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:46:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-08-31T22:46:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_2 2024-08-31T22:46:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:46:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_2 2024-08-31T22:46:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:46:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-08-31T22:46:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-08-31T22:46:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_3 2024-08-31T22:46:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-08-31T22:46:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:46:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-08-31T22:46:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_3 2024-08-31T22:46:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:46:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_3 2024-08-31T22:46:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:46:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-08-31T22:46:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-08-31T22:46:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_4 2024-08-31T22:46:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-08-31T22:46:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:46:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-08-31T22:46:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_4 2024-08-31T22:46:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:46:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_4 2024-08-31T22:46:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:46:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-08-31T22:46:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-08-31T22:46:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_5 2024-08-31T22:46:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-08-31T22:46:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:46:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-08-31T22:46:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_5 2024-08-31T22:46:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:46:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_5 2024-08-31T22:46:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:46:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-08-31T22:46:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-08-31T22:46:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_6 2024-08-31T22:46:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-08-31T22:46:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:46:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-08-31T22:46:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:15.429 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:46:15.428+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff16b60 2024-08-31T22:46:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_6 2024-08-31T22:46:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:46:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_6 2024-08-31T22:46:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:46:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-08-31T22:46:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-08-31T22:46:15.695 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:15.691+0000 7f6588f42640 -1 reset not still connected to 0x5620326af930 2024-08-31T22:46:15.695 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:15.691+0000 7f6588f42640 -1 reset not still connected to 0x562032a40680 2024-08-31T22:46:15.695 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:15.691+0000 7f6588f42640 -1 reset not still connected to 0x562032a40d00 2024-08-31T22:46:15.696 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:15.691+0000 7f6588f42640 -1 reset not still connected to 0x562037adec30 2024-08-31T22:46:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:15.699 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:15.695+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:46:15.699 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:15.699+0000 7f6588f42640 -1 reset not still connected to 0x562045dee270 2024-08-31T22:46:15.699 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:15.699+0000 7f6588f42640 -1 reset not still connected to 0x56204cad00d0 2024-08-31T22:46:15.700 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:15.699+0000 7f6588f42640 -1 reset not still connected to 0x562054033ee0 2024-08-31T22:46:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_7 2024-08-31T22:46:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-08-31T22:46:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:46:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-08-31T22:46:15.714 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:15.703+0000 7f6588f42640 -1 reset not still connected to 0x56205625cea0 2024-08-31T22:46:15.714 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:15.703+0000 7f6588f42640 -1 reset not still connected to 0x562057bb5040 2024-08-31T22:46:15.714 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:15.703+0000 7f6588f42640 -1 reset not still connected to 0x562058e9be10 2024-08-31T22:46:15.715 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:15.703+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:46:15.715 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:15.703+0000 7f6588f42640 -1 reset not still connected to 0x56205b8c0ea0 2024-08-31T22:46:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_7 2024-08-31T22:46:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:46:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_7 2024-08-31T22:46:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:46:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-08-31T22:46:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-08-31T22:46:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_8 2024-08-31T22:46:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-08-31T22:46:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:46:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-08-31T22:46:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_8 2024-08-31T22:46:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:46:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_8 2024-08-31T22:46:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:46:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-08-31T22:46:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-08-31T22:46:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_9 2024-08-31T22:46:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-08-31T22:46:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:46:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-08-31T22:46:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_9 2024-08-31T22:46:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:46:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_9 2024-08-31T22:46:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:46:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-08-31T22:46:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-08-31T22:46:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_10 2024-08-31T22:46:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-08-31T22:46:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:46:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-08-31T22:46:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_10 2024-08-31T22:46:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:46:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_10 2024-08-31T22:46:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:46:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-08-31T22:46:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-08-31T22:46:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:19.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:19.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:19.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_11 2024-08-31T22:46:19.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-08-31T22:46:19.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:19.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:19.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_11 2024-08-31T22:46:19.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:19.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-08-31T22:46:19.618 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:19.615+0000 7f6588f42640 -1 reset not still connected to 0x562045dee0d0 2024-08-31T22:46:19.619 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:19.615+0000 7f6588f42640 -1 reset not still connected to 0x562045dee270 2024-08-31T22:46:19.623 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:19.615+0000 7f6588f42640 -1 reset not still connected to 0x56204cad00d0 2024-08-31T22:46:19.624 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:19.619+0000 7f6588f42640 -1 reset not still connected to 0x562054033ee0 2024-08-31T22:46:19.624 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:19.619+0000 7f6588f42640 -1 reset not still connected to 0x562057bb5040 2024-08-31T22:46:19.626 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:19.623+0000 7f6588f42640 -1 reset not still connected to 0x562058e9be10 2024-08-31T22:46:19.626 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:19.623+0000 7f6588f42640 -1 reset not still connected to 0x56205949c410 2024-08-31T22:46:19.626 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:19.623+0000 7f6588f42640 -1 reset not still connected to 0x56205b8c0ea0 2024-08-31T22:46:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_11 2024-08-31T22:46:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_11 2024-08-31T22:46:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_11 2024-08-31T22:46:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_11 2024-08-31T22:46:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-08-31T22:46:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-08-31T22:46:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_12 2024-08-31T22:46:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-08-31T22:46:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_12 2024-08-31T22:46:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-08-31T22:46:20.811 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:46:20.811+0000 7f1a77b11640 -1 reset not still connected to 0x5649d2b4e4e0 2024-08-31T22:46:20.812 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:46:20.811+0000 7f1a77b11640 -1 reset not still connected to 0x5649dca20680 2024-08-31T22:46:20.817 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:46:20.815+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfc38000 2024-08-31T22:46:20.817 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:46:20.815+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:46:20.821 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:46:20.819+0000 7f1a77b11640 -1 reset not still connected to 0x5649e235e680 2024-08-31T22:46:20.822 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:46:20.819+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:46:20.826 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:46:20.823+0000 7f1a77b11640 -1 reset not still connected to 0x5649dca20680 2024-08-31T22:46:20.830 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:46:20.827+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfc38000 2024-08-31T22:46:20.830 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:46:20.827+0000 7f1a77b11640 -1 reset not still connected to 0x5649dfcd1450 2024-08-31T22:46:20.835 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:46:20.831+0000 7f1a77b11640 -1 reset not still connected to 0x5649e235e680 2024-08-31T22:46:20.835 INFO:tasks.ceph.cluster1.osd.1.smithi007.stderr:2024-08-31T22:46:20.831+0000 7f1a77b11640 -1 reset not still connected to 0x5649f8493790 2024-08-31T22:46:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_12 2024-08-31T22:46:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_12 2024-08-31T22:46:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_12 2024-08-31T22:46:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_12 2024-08-31T22:46:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-08-31T22:46:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-08-31T22:46:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_13 2024-08-31T22:46:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-08-31T22:46:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_13 2024-08-31T22:46:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-08-31T22:46:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:22.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_13 2024-08-31T22:46:22.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:22.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:22.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_13 2024-08-31T22:46:22.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:22.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_13 2024-08-31T22:46:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_13 2024-08-31T22:46:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-08-31T22:46:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-08-31T22:46:22.740 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:22.739+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:46:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:22.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:22.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:22.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_14 2024-08-31T22:46:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-08-31T22:46:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_14 2024-08-31T22:46:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-08-31T22:46:22.768 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:22.767+0000 7f6588f42640 -1 reset not still connected to 0x5620324664e0 2024-08-31T22:46:22.769 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:22.767+0000 7f6588f42640 -1 reset not still connected to 0x562037ec7ee0 2024-08-31T22:46:22.769 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:22.767+0000 7f6588f42640 -1 reset not still connected to 0x56203b44f930 2024-08-31T22:46:22.769 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:22.767+0000 7f6588f42640 -1 reset not still connected to 0x5620404e49c0 2024-08-31T22:46:22.769 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:22.767+0000 7f6588f42640 -1 reset not still connected to 0x562043c70b60 2024-08-31T22:46:22.769 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:22.767+0000 7f6588f42640 -1 reset not still connected to 0x562045a6a270 2024-08-31T22:46:22.769 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:22.767+0000 7f6588f42640 -1 reset not still connected to 0x562046a39380 2024-08-31T22:46:22.769 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:22.767+0000 7f6588f42640 -1 reset not still connected to 0x562046c83860 2024-08-31T22:46:22.769 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:22.767+0000 7f6588f42640 -1 reset not still connected to 0x562047502680 2024-08-31T22:46:22.769 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:22.767+0000 7f6588f42640 -1 reset not still connected to 0x562047503380 2024-08-31T22:46:22.769 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:22.767+0000 7f6588f42640 -1 reset not still connected to 0x562051047d40 2024-08-31T22:46:22.769 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:22.767+0000 7f6588f42640 -1 reset not still connected to 0x562051276340 2024-08-31T22:46:22.769 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:22.767+0000 7f6588f42640 -1 reset not still connected to 0x5620535de000 2024-08-31T22:46:22.769 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:22.767+0000 7f6588f42640 -1 reset not still connected to 0x5620546d80d0 2024-08-31T22:46:22.770 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:22.767+0000 7f6588f42640 -1 reset not still connected to 0x562058bed860 2024-08-31T22:46:22.772 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:22.771+0000 7f6588f42640 -1 reset not still connected to 0x562058e2bc70 2024-08-31T22:46:22.772 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:22.771+0000 7f6588f42640 -1 reset not still connected to 0x5620593a0f70 2024-08-31T22:46:22.772 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:22.771+0000 7f6588f42640 -1 reset not still connected to 0x56205d712b60 2024-08-31T22:46:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_14 2024-08-31T22:46:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_14 2024-08-31T22:46:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_14 2024-08-31T22:46:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_14 2024-08-31T22:46:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-08-31T22:46:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-08-31T22:46:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_15 2024-08-31T22:46:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-08-31T22:46:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_15 2024-08-31T22:46:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-08-31T22:46:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_15 2024-08-31T22:46:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_15 2024-08-31T22:46:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_15 2024-08-31T22:46:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_15 2024-08-31T22:46:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-08-31T22:46:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-08-31T22:46:24.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_16 2024-08-31T22:46:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-08-31T22:46:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_16 2024-08-31T22:46:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-08-31T22:46:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_16 2024-08-31T22:46:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_16 2024-08-31T22:46:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_16 2024-08-31T22:46:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_16 2024-08-31T22:46:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-08-31T22:46:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-08-31T22:46:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_17 2024-08-31T22:46:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-08-31T22:46:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:25.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_17 2024-08-31T22:46:25.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:25.726 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-08-31T22:46:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_17 2024-08-31T22:46:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_17 2024-08-31T22:46:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_17 2024-08-31T22:46:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_17 2024-08-31T22:46:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-08-31T22:46:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-08-31T22:46:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_18 2024-08-31T22:46:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-08-31T22:46:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_18 2024-08-31T22:46:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-08-31T22:46:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_18 2024-08-31T22:46:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_18 2024-08-31T22:46:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_18 2024-08-31T22:46:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_18 2024-08-31T22:46:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-08-31T22:46:27.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-08-31T22:46:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_19 2024-08-31T22:46:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-08-31T22:46:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_19 2024-08-31T22:46:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-08-31T22:46:28.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_19 2024-08-31T22:46:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_19 2024-08-31T22:46:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_19 2024-08-31T22:46:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_19 2024-08-31T22:46:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-08-31T22:46:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-08-31T22:46:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_20 2024-08-31T22:46:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-08-31T22:46:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_20 2024-08-31T22:46:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-08-31T22:46:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_20 2024-08-31T22:46:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_20 2024-08-31T22:46:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_20 2024-08-31T22:46:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_20 2024-08-31T22:46:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-08-31T22:46:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-08-31T22:46:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_21 2024-08-31T22:46:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-08-31T22:46:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_21 2024-08-31T22:46:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-08-31T22:46:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_21 2024-08-31T22:46:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_21 2024-08-31T22:46:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_21 2024-08-31T22:46:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_21 2024-08-31T22:46:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-08-31T22:46:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-08-31T22:46:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_22 2024-08-31T22:46:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-08-31T22:46:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_22 2024-08-31T22:46:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-08-31T22:46:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_22 2024-08-31T22:46:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_22 2024-08-31T22:46:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_22 2024-08-31T22:46:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_22 2024-08-31T22:46:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-08-31T22:46:31.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-08-31T22:46:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_23 2024-08-31T22:46:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-08-31T22:46:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_23 2024-08-31T22:46:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-08-31T22:46:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_23 2024-08-31T22:46:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_23 2024-08-31T22:46:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_23 2024-08-31T22:46:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_23 2024-08-31T22:46:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-08-31T22:46:32.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-08-31T22:46:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_24 2024-08-31T22:46:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-08-31T22:46:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_24 2024-08-31T22:46:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-08-31T22:46:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_24 2024-08-31T22:46:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_24 2024-08-31T22:46:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_24 2024-08-31T22:46:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_24 2024-08-31T22:46:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-08-31T22:46:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-08-31T22:46:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_25 2024-08-31T22:46:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-08-31T22:46:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_25 2024-08-31T22:46:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-08-31T22:46:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_25 2024-08-31T22:46:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_25 2024-08-31T22:46:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_25 2024-08-31T22:46:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_25 2024-08-31T22:46:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-08-31T22:46:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-08-31T22:46:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_26 2024-08-31T22:46:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-08-31T22:46:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_26 2024-08-31T22:46:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-08-31T22:46:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_26 2024-08-31T22:46:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_26 2024-08-31T22:46:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_26 2024-08-31T22:46:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_26 2024-08-31T22:46:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-08-31T22:46:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-08-31T22:46:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_27 2024-08-31T22:46:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-08-31T22:46:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_27 2024-08-31T22:46:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-08-31T22:46:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_27 2024-08-31T22:46:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_27 2024-08-31T22:46:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_27 2024-08-31T22:46:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_27 2024-08-31T22:46:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-08-31T22:46:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-08-31T22:46:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_28 2024-08-31T22:46:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-08-31T22:46:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_28 2024-08-31T22:46:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-08-31T22:46:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_28 2024-08-31T22:46:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_28 2024-08-31T22:46:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_28 2024-08-31T22:46:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_28 2024-08-31T22:46:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-08-31T22:46:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-08-31T22:46:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_29 2024-08-31T22:46:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-08-31T22:46:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_29 2024-08-31T22:46:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-08-31T22:46:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_29 2024-08-31T22:46:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_29 2024-08-31T22:46:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_29 2024-08-31T22:46:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_29 2024-08-31T22:46:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-08-31T22:46:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-08-31T22:46:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_30 2024-08-31T22:46:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-08-31T22:46:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_30 2024-08-31T22:46:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-08-31T22:46:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:39.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_30 2024-08-31T22:46:39.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_30 2024-08-31T22:46:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_30 2024-08-31T22:46:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_30 2024-08-31T22:46:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-08-31T22:46:39.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-08-31T22:46:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_31 2024-08-31T22:46:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-08-31T22:46:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_31 2024-08-31T22:46:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-08-31T22:46:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_31 2024-08-31T22:46:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_31 2024-08-31T22:46:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_31 2024-08-31T22:46:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_31 2024-08-31T22:46:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-08-31T22:46:40.789 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-08-31T22:46:41.015 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_32 2024-08-31T22:46:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-08-31T22:46:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_32 2024-08-31T22:46:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-08-31T22:46:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_32 2024-08-31T22:46:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_32 2024-08-31T22:46:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_32 2024-08-31T22:46:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_32 2024-08-31T22:46:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-08-31T22:46:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-08-31T22:46:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_33 2024-08-31T22:46:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-08-31T22:46:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_33 2024-08-31T22:46:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-08-31T22:46:42.048 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:46:42.047+0000 7f338ab71640 -1 reset not still connected to 0x555cea62d5f0 2024-08-31T22:46:42.048 INFO:tasks.ceph.cluster1.osd.2.smithi007.stderr:2024-08-31T22:46:42.047+0000 7f338ab71640 -1 reset not still connected to 0x555cfad9f040 2024-08-31T22:46:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_33 2024-08-31T22:46:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_33 2024-08-31T22:46:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_33 2024-08-31T22:46:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_33 2024-08-31T22:46:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-08-31T22:46:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-08-31T22:46:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_34 2024-08-31T22:46:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-08-31T22:46:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_34 2024-08-31T22:46:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-08-31T22:46:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_34 2024-08-31T22:46:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_34 2024-08-31T22:46:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_34 2024-08-31T22:46:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_34 2024-08-31T22:46:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-08-31T22:46:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-08-31T22:46:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_35 2024-08-31T22:46:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-08-31T22:46:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_35 2024-08-31T22:46:43.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:43.840 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-08-31T22:46:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_35 2024-08-31T22:46:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_35 2024-08-31T22:46:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_35 2024-08-31T22:46:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_35 2024-08-31T22:46:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-08-31T22:46:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-08-31T22:46:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_36 2024-08-31T22:46:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-08-31T22:46:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_36 2024-08-31T22:46:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-08-31T22:46:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_36 2024-08-31T22:46:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_36 2024-08-31T22:46:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_36 2024-08-31T22:46:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_36 2024-08-31T22:46:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-08-31T22:46:45.624 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-08-31T22:46:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_37 2024-08-31T22:46:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-08-31T22:46:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_37 2024-08-31T22:46:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-08-31T22:46:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_37 2024-08-31T22:46:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_37 2024-08-31T22:46:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_37 2024-08-31T22:46:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_37 2024-08-31T22:46:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-08-31T22:46:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-08-31T22:46:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_38 2024-08-31T22:46:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-08-31T22:46:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_38 2024-08-31T22:46:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-08-31T22:46:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_38 2024-08-31T22:46:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_38 2024-08-31T22:46:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_38 2024-08-31T22:46:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_38 2024-08-31T22:46:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-08-31T22:46:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-08-31T22:46:48.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_39 2024-08-31T22:46:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-08-31T22:46:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_39 2024-08-31T22:46:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-08-31T22:46:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_39 2024-08-31T22:46:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_39 2024-08-31T22:46:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_39 2024-08-31T22:46:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_39 2024-08-31T22:46:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-08-31T22:46:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-08-31T22:46:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:48.857 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_40 2024-08-31T22:46:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-08-31T22:46:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_40 2024-08-31T22:46:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-08-31T22:46:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_40 2024-08-31T22:46:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_40 2024-08-31T22:46:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_40 2024-08-31T22:46:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_40 2024-08-31T22:46:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-08-31T22:46:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-08-31T22:46:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_41 2024-08-31T22:46:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-08-31T22:46:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_41 2024-08-31T22:46:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-08-31T22:46:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_41 2024-08-31T22:46:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_41 2024-08-31T22:46:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_41 2024-08-31T22:46:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_41 2024-08-31T22:46:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-08-31T22:46:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-08-31T22:46:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_42 2024-08-31T22:46:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-08-31T22:46:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_42 2024-08-31T22:46:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-08-31T22:46:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_42 2024-08-31T22:46:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_42 2024-08-31T22:46:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_42 2024-08-31T22:46:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_42 2024-08-31T22:46:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-08-31T22:46:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-08-31T22:46:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_43 2024-08-31T22:46:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-08-31T22:46:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_43 2024-08-31T22:46:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-08-31T22:46:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_43 2024-08-31T22:46:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_43 2024-08-31T22:46:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_43 2024-08-31T22:46:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_43 2024-08-31T22:46:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-08-31T22:46:52.664 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-08-31T22:46:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_44 2024-08-31T22:46:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-08-31T22:46:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_44 2024-08-31T22:46:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-08-31T22:46:52.897 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:52.895+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:46:52.900 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:52.899+0000 7f6588f42640 -1 reset not still connected to 0x562030bca340 2024-08-31T22:46:52.900 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:52.899+0000 7f6588f42640 -1 reset not still connected to 0x56203d6a8ea0 2024-08-31T22:46:52.900 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:52.899+0000 7f6588f42640 -1 reset not still connected to 0x562051047d40 2024-08-31T22:46:52.907 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:52.903+0000 7f6588f42640 -1 reset not still connected to 0x562056716680 2024-08-31T22:46:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_44 2024-08-31T22:46:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_44 2024-08-31T22:46:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_44 2024-08-31T22:46:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_44 2024-08-31T22:46:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-08-31T22:46:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-08-31T22:46:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_45 2024-08-31T22:46:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-08-31T22:46:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_45 2024-08-31T22:46:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-08-31T22:46:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_45 2024-08-31T22:46:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_45 2024-08-31T22:46:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_45 2024-08-31T22:46:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_45 2024-08-31T22:46:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-08-31T22:46:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-08-31T22:46:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_46 2024-08-31T22:46:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-08-31T22:46:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_46 2024-08-31T22:46:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-08-31T22:46:55.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_46 2024-08-31T22:46:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_46 2024-08-31T22:46:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_46 2024-08-31T22:46:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_46 2024-08-31T22:46:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-08-31T22:46:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-08-31T22:46:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:55.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:55.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_47 2024-08-31T22:46:55.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-08-31T22:46:55.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:55.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:55.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_47 2024-08-31T22:46:55.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:55.874 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-08-31T22:46:56.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:56.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_47 2024-08-31T22:46:56.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:56.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:56.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_47 2024-08-31T22:46:56.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:56.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:56.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_47 2024-08-31T22:46:56.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:56.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:56.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_47 2024-08-31T22:46:56.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-08-31T22:46:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-08-31T22:46:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_48 2024-08-31T22:46:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-08-31T22:46:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_48 2024-08-31T22:46:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-08-31T22:46:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_48 2024-08-31T22:46:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_48 2024-08-31T22:46:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_48 2024-08-31T22:46:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_48 2024-08-31T22:46:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-08-31T22:46:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-08-31T22:46:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_49 2024-08-31T22:46:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-08-31T22:46:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_49 2024-08-31T22:46:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-08-31T22:46:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_49 2024-08-31T22:46:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_49 2024-08-31T22:46:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_49 2024-08-31T22:46:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_49 2024-08-31T22:46:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-08-31T22:46:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-08-31T22:46:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_50 2024-08-31T22:46:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-08-31T22:46:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_50 2024-08-31T22:46:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local snap=snap 2024-08-31T22:46:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-08-31T22:46:59.175 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:59.175+0000 7f6588f42640 -1 reset not still connected to 0x562025f71040 2024-08-31T22:46:59.176 INFO:tasks.ceph.cluster1.osd.0.smithi007.stderr:2024-08-31T22:46:59.175+0000 7f6588f42640 -1 reset not still connected to 0x5620469c7ad0 2024-08-31T22:46:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing snap: 100% complete...done. 2024-08-31T22:46:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_50 2024-08-31T22:46:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_50 2024-08-31T22:46:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:46:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:46:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_50 2024-08-31T22:46:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:46:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_50 2024-08-31T22:46:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-08-31T22:46:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-08-31T22:46:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-31T22:46:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ testlog 'TEST: image deletions should propagate' 2024-08-31T22:46:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ tee -a /tmp/tmp.iYfzsjLopX/rbd-mirror.test.log 2024-08-31T22:46:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ date '+%F %T' 2024-08-31T22:46:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo 2024-08-31 22:46:59 TEST: image deletions should propagate 2024-08-31T22:46:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:2024-08-31 22:46:59 TEST: image deletions should propagate 2024-08-31T22:46:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-08-31T22:46:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:46:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_count=0 2024-08-31T22:46:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:46:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local count 2024-08-31T22:46:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local last_count=0 2024-08-31T22:46:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ true 2024-08-31T22:46:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 1 40 2024-08-31T22:46:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in `seq 1 40` 2024-08-31T22:46:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 1 -ne 1 2024-08-31T22:46:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-08-31T22:46:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep 'images: ' 2024-08-31T22:46:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut '-d ' -f 2 2024-08-31T22:46:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: failed to query services: (13) Permission denied 2024-08-31T22:46:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ count=0 2024-08-31T22:46:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 0 = 0 2024-08-31T22:46:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-08-31T22:46:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:46:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:46:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state 2024-08-31T22:46:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 1 40 2024-08-31T22:46:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in `seq 1 40` 2024-08-31T22:46:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 1 -ne 1 2024-08-31T22:46:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-08-31T22:46:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep 'image health:' 2024-08-31T22:46:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cut '-d ' -f 3 2024-08-31T22:46:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: failed to query services: (13) Permission denied 2024-08-31T22:46:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ state=OK 2024-08-31T22:46:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test OK = ERROR 2024-08-31T22:46:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test OK = OK 2024-08-31T22:46:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 1 50 2024-08-31T22:46:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_1 2024-08-31T22:46:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-08-31T22:46:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:46:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:46:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:46:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:46:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:46:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:46:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_1 2024-08-31T22:46:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:46:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:46:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_1 2024-08-31T22:46:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-08-31T22:46:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:46:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-08-31T22:46:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:46:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:46:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:46:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-08-31T22:46:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:46:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:46:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:46:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:46:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:46:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:46:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_1 2024-08-31T22:46:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:46:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:46:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_1 2024-08-31T22:46:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-08-31T22:46:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:46:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-08-31T22:46:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:46:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:46:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:46:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:46:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:46:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_2 2024-08-31T22:46:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-08-31T22:46:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:46:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:46:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:46:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:46:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:46:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:46:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:46:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_2 2024-08-31T22:46:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:46:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:46:59.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_2 2024-08-31T22:46:59.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-08-31T22:46:59.930 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:00.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-08-31T22:47:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-08-31T22:47:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:47:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_2 2024-08-31T22:47:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:00.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_2 2024-08-31T22:47:00.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-08-31T22:47:00.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-08-31T22:47:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_3 2024-08-31T22:47:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-08-31T22:47:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:47:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_3 2024-08-31T22:47:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_3 2024-08-31T22:47:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-08-31T22:47:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-08-31T22:47:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-08-31T22:47:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:47:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:00.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_3 2024-08-31T22:47:00.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:00.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:00.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_3 2024-08-31T22:47:00.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-08-31T22:47:00.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-08-31T22:47:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_4 2024-08-31T22:47:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-08-31T22:47:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:47:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_4 2024-08-31T22:47:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_4 2024-08-31T22:47:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-08-31T22:47:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-08-31T22:47:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-08-31T22:47:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:47:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_4 2024-08-31T22:47:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_4 2024-08-31T22:47:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-08-31T22:47:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-08-31T22:47:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_5 2024-08-31T22:47:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-08-31T22:47:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:47:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_5 2024-08-31T22:47:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_5 2024-08-31T22:47:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-08-31T22:47:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-08-31T22:47:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:00.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-08-31T22:47:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:47:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_5 2024-08-31T22:47:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_5 2024-08-31T22:47:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-08-31T22:47:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-08-31T22:47:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_6 2024-08-31T22:47:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-08-31T22:47:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:47:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_6 2024-08-31T22:47:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_6 2024-08-31T22:47:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-08-31T22:47:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-08-31T22:47:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-08-31T22:47:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:47:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_6 2024-08-31T22:47:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_6 2024-08-31T22:47:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-08-31T22:47:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:01.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-08-31T22:47:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_7 2024-08-31T22:47:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-08-31T22:47:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:47:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_7 2024-08-31T22:47:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_7 2024-08-31T22:47:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-08-31T22:47:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-08-31T22:47:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-08-31T22:47:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:47:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_7 2024-08-31T22:47:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_7 2024-08-31T22:47:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-08-31T22:47:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-08-31T22:47:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_8 2024-08-31T22:47:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-08-31T22:47:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:47:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_8 2024-08-31T22:47:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_8 2024-08-31T22:47:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-08-31T22:47:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-08-31T22:47:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-08-31T22:47:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:47:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_8 2024-08-31T22:47:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_8 2024-08-31T22:47:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-08-31T22:47:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-08-31T22:47:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_9 2024-08-31T22:47:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-08-31T22:47:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:47:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_9 2024-08-31T22:47:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_9 2024-08-31T22:47:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-08-31T22:47:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-08-31T22:47:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-08-31T22:47:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:47:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_9 2024-08-31T22:47:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_9 2024-08-31T22:47:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-08-31T22:47:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-08-31T22:47:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_10 2024-08-31T22:47:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-08-31T22:47:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:47:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_10 2024-08-31T22:47:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_10 2024-08-31T22:47:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-08-31T22:47:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-08-31T22:47:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:02.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-08-31T22:47:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:47:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_10 2024-08-31T22:47:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_10 2024-08-31T22:47:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-08-31T22:47:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-08-31T22:47:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_11 2024-08-31T22:47:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-08-31T22:47:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_11 2024-08-31T22:47:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_11 2024-08-31T22:47:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_11 2024-08-31T22:47:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-08-31T22:47:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-08-31T22:47:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-08-31T22:47:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_11 2024-08-31T22:47:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_11 2024-08-31T22:47:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_11 2024-08-31T22:47:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-08-31T22:47:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-08-31T22:47:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_12 2024-08-31T22:47:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-08-31T22:47:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_12 2024-08-31T22:47:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_12 2024-08-31T22:47:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_12 2024-08-31T22:47:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-08-31T22:47:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-08-31T22:47:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-08-31T22:47:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_12 2024-08-31T22:47:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_12 2024-08-31T22:47:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_12 2024-08-31T22:47:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-08-31T22:47:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-08-31T22:47:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_13 2024-08-31T22:47:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-08-31T22:47:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_13 2024-08-31T22:47:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_13 2024-08-31T22:47:02.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:02.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:02.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_13 2024-08-31T22:47:02.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-08-31T22:47:02.772 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:02.808 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-08-31T22:47:02.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:02.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:02.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:02.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-08-31T22:47:02.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:02.914 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_13 2024-08-31T22:47:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_13 2024-08-31T22:47:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_13 2024-08-31T22:47:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-08-31T22:47:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-08-31T22:47:02.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:02.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_14 2024-08-31T22:47:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-08-31T22:47:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_14 2024-08-31T22:47:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_14 2024-08-31T22:47:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_14 2024-08-31T22:47:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-08-31T22:47:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-08-31T22:47:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-08-31T22:47:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_14 2024-08-31T22:47:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_14 2024-08-31T22:47:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_14 2024-08-31T22:47:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-08-31T22:47:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-08-31T22:47:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_15 2024-08-31T22:47:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-08-31T22:47:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_15 2024-08-31T22:47:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_15 2024-08-31T22:47:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_15 2024-08-31T22:47:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-08-31T22:47:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-08-31T22:47:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:03.312 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-08-31T22:47:03.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:03.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:03.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_15 2024-08-31T22:47:03.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:03.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:03.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:03.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_15 2024-08-31T22:47:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_15 2024-08-31T22:47:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-08-31T22:47:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-08-31T22:47:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_16 2024-08-31T22:47:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-08-31T22:47:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_16 2024-08-31T22:47:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_16 2024-08-31T22:47:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_16 2024-08-31T22:47:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-08-31T22:47:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:03.400 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-08-31T22:47:03.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:03.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:03.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-08-31T22:47:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_16 2024-08-31T22:47:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_16 2024-08-31T22:47:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_16 2024-08-31T22:47:03.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-08-31T22:47:03.507 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:03.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-08-31T22:47:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_17 2024-08-31T22:47:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-08-31T22:47:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_17 2024-08-31T22:47:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_17 2024-08-31T22:47:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_17 2024-08-31T22:47:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-08-31T22:47:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-08-31T22:47:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-08-31T22:47:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_17 2024-08-31T22:47:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_17 2024-08-31T22:47:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_17 2024-08-31T22:47:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-08-31T22:47:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-08-31T22:47:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_18 2024-08-31T22:47:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-08-31T22:47:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_18 2024-08-31T22:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_18 2024-08-31T22:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_18 2024-08-31T22:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-08-31T22:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-08-31T22:47:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-08-31T22:47:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_18 2024-08-31T22:47:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_18 2024-08-31T22:47:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_18 2024-08-31T22:47:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-08-31T22:47:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-08-31T22:47:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_19 2024-08-31T22:47:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-08-31T22:47:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_19 2024-08-31T22:47:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_19 2024-08-31T22:47:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_19 2024-08-31T22:47:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-08-31T22:47:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-08-31T22:47:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-08-31T22:47:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_19 2024-08-31T22:47:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_19 2024-08-31T22:47:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_19 2024-08-31T22:47:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-08-31T22:47:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-08-31T22:47:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_20 2024-08-31T22:47:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-08-31T22:47:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_20 2024-08-31T22:47:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_20 2024-08-31T22:47:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_20 2024-08-31T22:47:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-08-31T22:47:04.150 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-08-31T22:47:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-08-31T22:47:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_20 2024-08-31T22:47:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_20 2024-08-31T22:47:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_20 2024-08-31T22:47:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-08-31T22:47:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-08-31T22:47:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_21 2024-08-31T22:47:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-08-31T22:47:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_21 2024-08-31T22:47:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_21 2024-08-31T22:47:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_21 2024-08-31T22:47:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-08-31T22:47:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-08-31T22:47:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-08-31T22:47:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_21 2024-08-31T22:47:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_21 2024-08-31T22:47:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_21 2024-08-31T22:47:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-08-31T22:47:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-08-31T22:47:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_22 2024-08-31T22:47:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-08-31T22:47:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_22 2024-08-31T22:47:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_22 2024-08-31T22:47:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_22 2024-08-31T22:47:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-08-31T22:47:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-08-31T22:47:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-08-31T22:47:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_22 2024-08-31T22:47:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_22 2024-08-31T22:47:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_22 2024-08-31T22:47:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-08-31T22:47:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-08-31T22:47:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_23 2024-08-31T22:47:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-08-31T22:47:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_23 2024-08-31T22:47:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_23 2024-08-31T22:47:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_23 2024-08-31T22:47:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-08-31T22:47:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-08-31T22:47:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-08-31T22:47:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_23 2024-08-31T22:47:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_23 2024-08-31T22:47:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_23 2024-08-31T22:47:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-08-31T22:47:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-08-31T22:47:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_24 2024-08-31T22:47:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-08-31T22:47:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_24 2024-08-31T22:47:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_24 2024-08-31T22:47:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_24 2024-08-31T22:47:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-08-31T22:47:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-08-31T22:47:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-08-31T22:47:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_24 2024-08-31T22:47:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_24 2024-08-31T22:47:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_24 2024-08-31T22:47:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-08-31T22:47:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-08-31T22:47:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_25 2024-08-31T22:47:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-08-31T22:47:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_25 2024-08-31T22:47:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_25 2024-08-31T22:47:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_25 2024-08-31T22:47:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-08-31T22:47:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-08-31T22:47:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-08-31T22:47:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_25 2024-08-31T22:47:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_25 2024-08-31T22:47:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_25 2024-08-31T22:47:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-08-31T22:47:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:05.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-08-31T22:47:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_26 2024-08-31T22:47:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-08-31T22:47:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_26 2024-08-31T22:47:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_26 2024-08-31T22:47:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_26 2024-08-31T22:47:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-08-31T22:47:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-08-31T22:47:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:05.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-08-31T22:47:05.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:05.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:05.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_26 2024-08-31T22:47:05.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:05.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:05.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:05.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_26 2024-08-31T22:47:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_26 2024-08-31T22:47:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-08-31T22:47:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-08-31T22:47:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_27 2024-08-31T22:47:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-08-31T22:47:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_27 2024-08-31T22:47:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_27 2024-08-31T22:47:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_27 2024-08-31T22:47:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-08-31T22:47:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-08-31T22:47:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-08-31T22:47:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_27 2024-08-31T22:47:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_27 2024-08-31T22:47:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_27 2024-08-31T22:47:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-08-31T22:47:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-08-31T22:47:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_28 2024-08-31T22:47:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-08-31T22:47:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_28 2024-08-31T22:47:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_28 2024-08-31T22:47:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_28 2024-08-31T22:47:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-08-31T22:47:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:05.777 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-08-31T22:47:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-08-31T22:47:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_28 2024-08-31T22:47:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_28 2024-08-31T22:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_28 2024-08-31T22:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-08-31T22:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-08-31T22:47:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_29 2024-08-31T22:47:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-08-31T22:47:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_29 2024-08-31T22:47:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_29 2024-08-31T22:47:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_29 2024-08-31T22:47:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-08-31T22:47:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-08-31T22:47:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-08-31T22:47:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_29 2024-08-31T22:47:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_29 2024-08-31T22:47:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_29 2024-08-31T22:47:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-08-31T22:47:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-08-31T22:47:06.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_30 2024-08-31T22:47:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-08-31T22:47:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_30 2024-08-31T22:47:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_30 2024-08-31T22:47:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_30 2024-08-31T22:47:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-08-31T22:47:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-08-31T22:47:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-08-31T22:47:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_30 2024-08-31T22:47:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_30 2024-08-31T22:47:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_30 2024-08-31T22:47:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-08-31T22:47:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-08-31T22:47:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_31 2024-08-31T22:47:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-08-31T22:47:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_31 2024-08-31T22:47:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_31 2024-08-31T22:47:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_31 2024-08-31T22:47:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-08-31T22:47:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-08-31T22:47:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-08-31T22:47:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_31 2024-08-31T22:47:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_31 2024-08-31T22:47:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_31 2024-08-31T22:47:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-08-31T22:47:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-08-31T22:47:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_32 2024-08-31T22:47:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-08-31T22:47:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_32 2024-08-31T22:47:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_32 2024-08-31T22:47:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_32 2024-08-31T22:47:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-08-31T22:47:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:06.569 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-08-31T22:47:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-08-31T22:47:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_32 2024-08-31T22:47:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_32 2024-08-31T22:47:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_32 2024-08-31T22:47:06.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-08-31T22:47:06.677 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-08-31T22:47:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_33 2024-08-31T22:47:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-08-31T22:47:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_33 2024-08-31T22:47:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_33 2024-08-31T22:47:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_33 2024-08-31T22:47:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-08-31T22:47:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-08-31T22:47:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-08-31T22:47:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_33 2024-08-31T22:47:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_33 2024-08-31T22:47:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_33 2024-08-31T22:47:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-08-31T22:47:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-08-31T22:47:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_34 2024-08-31T22:47:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-08-31T22:47:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_34 2024-08-31T22:47:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_34 2024-08-31T22:47:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_34 2024-08-31T22:47:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-08-31T22:47:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-08-31T22:47:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-08-31T22:47:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_34 2024-08-31T22:47:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_34 2024-08-31T22:47:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_34 2024-08-31T22:47:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-08-31T22:47:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:07.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-08-31T22:47:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_35 2024-08-31T22:47:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-08-31T22:47:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_35 2024-08-31T22:47:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_35 2024-08-31T22:47:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_35 2024-08-31T22:47:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-08-31T22:47:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-08-31T22:47:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-08-31T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_35 2024-08-31T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_35 2024-08-31T22:47:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_35 2024-08-31T22:47:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-08-31T22:47:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:07.342 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-08-31T22:47:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_36 2024-08-31T22:47:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-08-31T22:47:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_36 2024-08-31T22:47:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_36 2024-08-31T22:47:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_36 2024-08-31T22:47:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-08-31T22:47:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-08-31T22:47:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-08-31T22:47:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_36 2024-08-31T22:47:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_36 2024-08-31T22:47:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_36 2024-08-31T22:47:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-08-31T22:47:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-08-31T22:47:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_37 2024-08-31T22:47:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-08-31T22:47:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_37 2024-08-31T22:47:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_37 2024-08-31T22:47:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_37 2024-08-31T22:47:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-08-31T22:47:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-08-31T22:47:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:07.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-08-31T22:47:07.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:07.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_37 2024-08-31T22:47:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_37 2024-08-31T22:47:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_37 2024-08-31T22:47:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-08-31T22:47:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-08-31T22:47:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_38 2024-08-31T22:47:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-08-31T22:47:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_38 2024-08-31T22:47:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_38 2024-08-31T22:47:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_38 2024-08-31T22:47:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-08-31T22:47:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-08-31T22:47:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-08-31T22:47:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_38 2024-08-31T22:47:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_38 2024-08-31T22:47:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_38 2024-08-31T22:47:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-08-31T22:47:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-08-31T22:47:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_39 2024-08-31T22:47:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-08-31T22:47:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_39 2024-08-31T22:47:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_39 2024-08-31T22:47:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_39 2024-08-31T22:47:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-08-31T22:47:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-08-31T22:47:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-08-31T22:47:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_39 2024-08-31T22:47:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_39 2024-08-31T22:47:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_39 2024-08-31T22:47:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-08-31T22:47:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-08-31T22:47:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_40 2024-08-31T22:47:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-08-31T22:47:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_40 2024-08-31T22:47:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_40 2024-08-31T22:47:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_40 2024-08-31T22:47:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-08-31T22:47:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-08-31T22:47:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-08-31T22:47:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_40 2024-08-31T22:47:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_40 2024-08-31T22:47:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_40 2024-08-31T22:47:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-08-31T22:47:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-08-31T22:47:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_41 2024-08-31T22:47:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-08-31T22:47:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_41 2024-08-31T22:47:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_41 2024-08-31T22:47:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_41 2024-08-31T22:47:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-08-31T22:47:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-08-31T22:47:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-08-31T22:47:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_41 2024-08-31T22:47:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_41 2024-08-31T22:47:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_41 2024-08-31T22:47:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-08-31T22:47:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-08-31T22:47:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_42 2024-08-31T22:47:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-08-31T22:47:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_42 2024-08-31T22:47:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_42 2024-08-31T22:47:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_42 2024-08-31T22:47:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-08-31T22:47:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-08-31T22:47:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-08-31T22:47:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_42 2024-08-31T22:47:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_42 2024-08-31T22:47:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_42 2024-08-31T22:47:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-08-31T22:47:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-08-31T22:47:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_43 2024-08-31T22:47:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-08-31T22:47:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_43 2024-08-31T22:47:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_43 2024-08-31T22:47:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_43 2024-08-31T22:47:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-08-31T22:47:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-08-31T22:47:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:08.887 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-08-31T22:47:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_43 2024-08-31T22:47:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:08.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_43 2024-08-31T22:47:08.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:08.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:08.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_43 2024-08-31T22:47:08.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-08-31T22:47:08.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-08-31T22:47:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_44 2024-08-31T22:47:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-08-31T22:47:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_44 2024-08-31T22:47:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_44 2024-08-31T22:47:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_44 2024-08-31T22:47:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-08-31T22:47:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-08-31T22:47:08.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:08.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:08.984 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-08-31T22:47:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_44 2024-08-31T22:47:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_44 2024-08-31T22:47:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_44 2024-08-31T22:47:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-08-31T22:47:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-08-31T22:47:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_45 2024-08-31T22:47:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-08-31T22:47:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_45 2024-08-31T22:47:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_45 2024-08-31T22:47:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_45 2024-08-31T22:47:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-08-31T22:47:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-08-31T22:47:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-08-31T22:47:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_45 2024-08-31T22:47:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_45 2024-08-31T22:47:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_45 2024-08-31T22:47:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-08-31T22:47:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-08-31T22:47:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_46 2024-08-31T22:47:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-08-31T22:47:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_46 2024-08-31T22:47:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_46 2024-08-31T22:47:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_46 2024-08-31T22:47:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-08-31T22:47:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-08-31T22:47:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-08-31T22:47:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_46 2024-08-31T22:47:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_46 2024-08-31T22:47:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_46 2024-08-31T22:47:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-08-31T22:47:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-08-31T22:47:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_47 2024-08-31T22:47:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-08-31T22:47:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_47 2024-08-31T22:47:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_47 2024-08-31T22:47:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_47 2024-08-31T22:47:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-08-31T22:47:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-08-31T22:47:09.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:09.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:09.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-08-31T22:47:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_47 2024-08-31T22:47:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_47 2024-08-31T22:47:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_47 2024-08-31T22:47:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-08-31T22:47:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-08-31T22:47:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_48 2024-08-31T22:47:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-08-31T22:47:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_48 2024-08-31T22:47:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_48 2024-08-31T22:47:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_48 2024-08-31T22:47:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-08-31T22:47:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-08-31T22:47:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-08-31T22:47:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_48 2024-08-31T22:47:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_48 2024-08-31T22:47:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_48 2024-08-31T22:47:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-08-31T22:47:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-08-31T22:47:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_49 2024-08-31T22:47:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-08-31T22:47:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_49 2024-08-31T22:47:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_49 2024-08-31T22:47:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_49 2024-08-31T22:47:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-08-31T22:47:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-08-31T22:47:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-08-31T22:47:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_49 2024-08-31T22:47:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_49 2024-08-31T22:47:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_49 2024-08-31T22:47:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-08-31T22:47:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-08-31T22:47:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-08-31T22:47:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_50 2024-08-31T22:47:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-08-31T22:47:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_50 2024-08-31T22:47:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_50 2024-08-31T22:47:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_50 2024-08-31T22:47:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-08-31T22:47:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-08-31T22:47:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-08-31T22:47:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_50 2024-08-31T22:47:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_50 2024-08-31T22:47:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_50 2024-08-31T22:47:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-08-31T22:47:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-08-31T22:47:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-08-31T22:47:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ tee -a /tmp/tmp.iYfzsjLopX/rbd-mirror.test.log 2024-08-31T22:47:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ date '+%F %T' 2024-08-31T22:47:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo 2024-08-31 22:47:10 TEST: delete images during bootstrap 2024-08-31T22:47:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:2024-08-31 22:47:10 TEST: delete images during bootstrap 2024-08-31T22:47:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-08-31T22:47:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=image 2024-08-31T22:47:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-08-31T22:47:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:note: changing mirroring mode from pool to image 2024-08-31T22:47:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-08-31T22:47:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:47:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=image 2024-08-31T22:47:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-08-31T22:47:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:note: changing mirroring mode from pool to image 2024-08-31T22:47:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ start_mirror cluster1 2024-08-31T22:47:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance 2024-08-31T22:47:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set_cluster_instance cluster1 cluster instance 2024-08-31T22:47:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local val=cluster1 2024-08-31T22:47:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster_var_name=cluster 2024-08-31T22:47:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local instance_var_name=instance 2024-08-31T22:47:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cluster=cluster1 2024-08-31T22:47:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=cluster1 2024-08-31T22:47:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' cluster1 = cluster1 ']' 2024-08-31T22:47:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ instance=0 2024-08-31T22:47:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval cluster=cluster1 2024-08-31T22:47:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:47:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ eval instance=0 2024-08-31T22:47:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:47:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-08-31T22:47:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:2024-08-31T22:47:10.523+0000 7f3c35212880 -1 open: failed to lock pidfile /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid because another process locked it': (11) Resource temporarily unavailable 2024-08-31T22:47:10.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=test 2024-08-31T22:47:10.525 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 1 10 2024-08-31T22:47:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:47:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_1 2024-08-31T22:47:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_1 512M 2024-08-31T22:47:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:47:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:47:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:47:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:47:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:47:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:47:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 512M ']' 2024-08-31T22:47:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=512M 2024-08-31T22:47:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:47:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_1 2024-08-31T22:47:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ enable_mirror cluster2 mirror image_1 2024-08-31T22:47:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:47:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:47:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:47:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 journal 2024-08-31T22:47:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring enabled 2024-08-31T22:47:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-08-31T22:47:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_1': 2024-08-31T22:47:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:47:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:47:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 0 2024-08-31T22:47:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 3bf684c70a4 2024-08-31T22:47:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.3bf684c70a4 2024-08-31T22:47:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:47:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:47:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:47:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:47:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:47:10 2024 2024-08-31T22:47:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:47:10 2024 2024-08-31T22:47:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:47:10 2024 2024-08-31T22:47:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 3bf684c70a4 2024-08-31T22:47:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:47:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:47:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: c6567716-dafb-4c89-a749-af0d07761b12 2024-08-31T22:47:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: true 2024-08-31T22:47:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_1 2024-08-31T22:47:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:47:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:47:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:47:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=12 2024-08-31T22:47:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:47:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:47:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 9385984 bytes 2024-08-31T22:47:13.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 59012096 bytes 2024-08-31T22:47:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 86637568 bytes 2024-08-31T22:47:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 144865280 bytes 2024-08-31T22:47:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:500 IOs, 174406656 bytes 2024-08-31T22:47:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:47:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:47:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:47:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-08-31T22:47:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:47:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=present 2024-08-31T22:47:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_1 2024-08-31T22:47:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_1 2024-08-31T22:47:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-08-31T22:47:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id=10a8f3c72d0b 2024-08-31T22:47:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_1 present 10a8f3c72d0b 2024-08-31T22:47:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:47:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=present 2024-08-31T22:47:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id=10a8f3c72d0b 2024-08-31T22:47:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_1 2024-08-31T22:47:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_1 2024-08-31T22:47:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-08-31T22:47:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id=10a8f3c72d0b 2024-08-31T22:47:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 10a8f3c72d0b 2024-08-31T22:47:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -z 10a8f3c72d0b -o 10a8f3c72d0b = 10a8f3c72d0b 2024-08-31T22:47:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=present 2024-08-31T22:47:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test present = present 2024-08-31T22:47:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ disable_mirror cluster2 mirror image_1 2024-08-31T22:47:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:47:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:22.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:47:22.935 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-08-31T22:47:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring disabled 2024-08-31T22:47:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-08-31T22:47:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:47:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:47:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:47:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_1 2024-08-31T22:47:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_1 2024-08-31T22:47:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-08-31T22:47:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-08-31T22:47:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:47:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:47:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:47:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-08-31T22:47:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:47:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:47:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:47:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:47:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:47:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:47:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_1 2024-08-31T22:47:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:47:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:47:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_1 2024-08-31T22:47:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-08-31T22:47:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:47:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-08-31T22:47:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:47:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:47:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:47:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ purge_snapshots cluster2 mirror image_1 2024-08-31T22:47:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:47:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:47:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-08-31T22:47:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_1 2024-08-31T22:47:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:47:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:47:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:47:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:47:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_1 2024-08-31T22:47:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:47:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_1 2024-08-31T22:47:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-08-31T22:47:53.807 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-08-31T22:47:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-08-31T22:47:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:47:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:47:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_2 2024-08-31T22:47:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_2 512M 2024-08-31T22:47:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:47:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:47:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:47:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:47:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:47:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:47:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 512M ']' 2024-08-31T22:47:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=512M 2024-08-31T22:47:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:47:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_2 2024-08-31T22:47:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ enable_mirror cluster2 mirror image_2 2024-08-31T22:47:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:47:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:47:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:47:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 journal 2024-08-31T22:47:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring enabled 2024-08-31T22:47:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-08-31T22:47:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_2': 2024-08-31T22:47:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:47:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 0 2024-08-31T22:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 3c0ec040b133 2024-08-31T22:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.3c0ec040b133 2024-08-31T22:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:47:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:47:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:47:54 2024 2024-08-31T22:47:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:47:54 2024 2024-08-31T22:47:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:47:54 2024 2024-08-31T22:47:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 3c0ec040b133 2024-08-31T22:47:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:47:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:47:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: ca7a10a3-cbe1-441e-bf70-1933556545f8 2024-08-31T22:47:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: true 2024-08-31T22:47:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_2 2024-08-31T22:47:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:47:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:47:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:47:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:47:54.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=7 2024-08-31T22:47:54.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:47:54.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:47:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 35652608 bytes 2024-08-31T22:47:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 91144192 bytes 2024-08-31T22:48:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:48:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:48:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:48:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:48:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-08-31T22:48:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:48:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:48:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:48:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=present 2024-08-31T22:48:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:48:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:48:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:48:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_2 2024-08-31T22:48:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:48:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:48:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_2 2024-08-31T22:48:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-08-31T22:48:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:48:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id=10a87a03539d 2024-08-31T22:48:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:48:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:48:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_2 present 10a87a03539d 2024-08-31T22:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=present 2024-08-31T22:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id=10a87a03539d 2024-08-31T22:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:48:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_2 2024-08-31T22:48:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:48:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:48:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_2 2024-08-31T22:48:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-08-31T22:48:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:48:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id=10a87a03539d 2024-08-31T22:48:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 10a87a03539d 2024-08-31T22:48:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -z 10a87a03539d -o 10a87a03539d = 10a87a03539d 2024-08-31T22:48:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=present 2024-08-31T22:48:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test present = present 2024-08-31T22:48:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:48:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ disable_mirror cluster2 mirror image_2 2024-08-31T22:48:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:48:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:48:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:48:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-08-31T22:48:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring disabled 2024-08-31T22:48:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-08-31T22:48:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:48:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:48:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:48:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:48:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:48:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:48:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:48:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_2 2024-08-31T22:48:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:48:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:48:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_2 2024-08-31T22:48:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-08-31T22:48:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:48:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-08-31T22:48:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:48:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:48:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:48:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-08-31T22:48:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:48:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:48:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:48:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:48:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:48:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:48:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:48:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_2 2024-08-31T22:48:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:48:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:48:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_2 2024-08-31T22:48:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-08-31T22:48:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:48:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-08-31T22:48:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:48:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:48:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:48:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:48:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ purge_snapshots cluster2 mirror image_2 2024-08-31T22:48:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:48:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:48:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:48:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-08-31T22:48:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_2 2024-08-31T22:48:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:48:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:48:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:48:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:48:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:48:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_2 2024-08-31T22:48:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:48:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:48:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_2 2024-08-31T22:48:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-08-31T22:48:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-08-31T22:48:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-08-31T22:48:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:48:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:48:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_3 2024-08-31T22:48:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_3 512M 2024-08-31T22:48:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:48:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:48:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:48:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:48:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:48:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:48:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:48:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 512M ']' 2024-08-31T22:48:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=512M 2024-08-31T22:48:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:48:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_3 2024-08-31T22:48:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ enable_mirror cluster2 mirror image_3 2024-08-31T22:48:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:48:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:48:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:48:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:48:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 journal 2024-08-31T22:48:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring enabled 2024-08-31T22:48:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-08-31T22:48:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_3': 2024-08-31T22:48:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:48:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:48:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 0 2024-08-31T22:48:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 3c262da90c86 2024-08-31T22:48:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.3c262da90c86 2024-08-31T22:48:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:48:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:48:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:48:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:48:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:48:32 2024 2024-08-31T22:48:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:48:32 2024 2024-08-31T22:48:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:48:32 2024 2024-08-31T22:48:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 3c262da90c86 2024-08-31T22:48:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:48:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:48:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: bbbbd715-e773-430f-9cf7-586378629131 2024-08-31T22:48:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: true 2024-08-31T22:48:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_3 2024-08-31T22:48:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:48:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:48:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:48:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:48:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=12 2024-08-31T22:48:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:48:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:48:34.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 46072832 bytes 2024-08-31T22:48:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 75210752 bytes 2024-08-31T22:48:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 91646976 bytes 2024-08-31T22:48:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 92266496 bytes 2024-08-31T22:48:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:500 IOs, 114279424 bytes 2024-08-31T22:48:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:600 IOs, 145126400 bytes 2024-08-31T22:48:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:48:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:48:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:48:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:48:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-08-31T22:48:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:48:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:48:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:48:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=present 2024-08-31T22:48:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:48:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:48:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:48:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_3 2024-08-31T22:48:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:48:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:48:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_3 2024-08-31T22:48:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:48:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-08-31T22:48:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id=10a8f9d636df 2024-08-31T22:48:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:48:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:48:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_3 present 10a8f9d636df 2024-08-31T22:48:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:48:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:48:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:48:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=present 2024-08-31T22:48:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id=10a8f9d636df 2024-08-31T22:48:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:48:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:48:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_3 2024-08-31T22:48:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:48:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:48:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_3 2024-08-31T22:48:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-08-31T22:48:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:48:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id=10a8f9d636df 2024-08-31T22:48:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 10a8f9d636df 2024-08-31T22:48:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -z 10a8f9d636df -o 10a8f9d636df = 10a8f9d636df 2024-08-31T22:48:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=present 2024-08-31T22:48:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test present = present 2024-08-31T22:48:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:48:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ disable_mirror cluster2 mirror image_3 2024-08-31T22:48:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:48:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:48:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:48:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-08-31T22:49:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring disabled 2024-08-31T22:49:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-08-31T22:49:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:49:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:49:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:49:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:49:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:49:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:49:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:49:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_3 2024-08-31T22:49:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:49:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:49:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_3 2024-08-31T22:49:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-08-31T22:49:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:49:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-08-31T22:49:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:49:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:49:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-08-31T22:49:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:49:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:49:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:49:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:49:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:49:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:49:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:49:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_3 2024-08-31T22:49:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:49:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:49:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_3 2024-08-31T22:49:12.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-08-31T22:49:12.730 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:49:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-08-31T22:49:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:49:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:49:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:49:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:49:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ purge_snapshots cluster2 mirror image_3 2024-08-31T22:49:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:49:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:49:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:49:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-08-31T22:49:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_3 2024-08-31T22:49:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:49:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:49:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:49:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:49:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:49:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_3 2024-08-31T22:49:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:49:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:49:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_3 2024-08-31T22:49:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-08-31T22:49:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-08-31T22:49:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-08-31T22:49:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:49:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:49:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_4 2024-08-31T22:49:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_4 512M 2024-08-31T22:49:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:49:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:49:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:49:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:49:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:49:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:49:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:49:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 512M ']' 2024-08-31T22:49:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=512M 2024-08-31T22:49:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:49:13.136 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_4 2024-08-31T22:49:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ enable_mirror cluster2 mirror image_4 2024-08-31T22:49:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:49:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:49:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:49:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:49:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 journal 2024-08-31T22:49:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring enabled 2024-08-31T22:49:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-08-31T22:49:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_4': 2024-08-31T22:49:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:49:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:49:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 0 2024-08-31T22:49:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 3c3e11c3055d 2024-08-31T22:49:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.3c3e11c3055d 2024-08-31T22:49:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:49:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:49:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:49:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:49:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:49:13 2024 2024-08-31T22:49:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:49:13 2024 2024-08-31T22:49:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:49:13 2024 2024-08-31T22:49:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 3c3e11c3055d 2024-08-31T22:49:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:49:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:49:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: fb09d83c-530c-470c-b778-1237ef24fa78 2024-08-31T22:49:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: true 2024-08-31T22:49:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_4 2024-08-31T22:49:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:49:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:49:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:49:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:49:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=11 2024-08-31T22:49:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:49:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:49:13.581 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 21830656 bytes 2024-08-31T22:49:14.780 INFO:tasks.ceph.cluster2.osd.0.smithi032.stderr:2024-08-31T22:49:14.780+0000 7ff273ffe640 -1 reset not still connected to 0x561938dc8a90 2024-08-31T22:49:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 121303040 bytes 2024-08-31T22:49:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 150693888 bytes 2024-08-31T22:49:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:400 IOs, 157880320 bytes 2024-08-31T22:49:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:49:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:49:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:49:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:49:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-08-31T22:49:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:49:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:49:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:49:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=present 2024-08-31T22:49:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:49:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:49:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:49:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_4 2024-08-31T22:49:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:49:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:49:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_4 2024-08-31T22:49:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-08-31T22:49:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:49:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id=10a8e82eaeb8 2024-08-31T22:49:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:49:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:49:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_4 present 10a8e82eaeb8 2024-08-31T22:49:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:49:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:49:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:49:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=present 2024-08-31T22:49:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id=10a8e82eaeb8 2024-08-31T22:49:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:49:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:49:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_4 2024-08-31T22:49:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:49:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:49:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_4 2024-08-31T22:49:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-08-31T22:49:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:49:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id=10a8e82eaeb8 2024-08-31T22:49:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 10a8e82eaeb8 2024-08-31T22:49:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -z 10a8e82eaeb8 -o 10a8e82eaeb8 = 10a8e82eaeb8 2024-08-31T22:49:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=present 2024-08-31T22:49:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test present = present 2024-08-31T22:49:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:49:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ disable_mirror cluster2 mirror image_4 2024-08-31T22:49:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:49:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:49:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:49:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-08-31T22:49:54.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring disabled 2024-08-31T22:49:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-08-31T22:49:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:49:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:49:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:49:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:49:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:49:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:49:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:49:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_4 2024-08-31T22:49:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:49:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:49:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_4 2024-08-31T22:49:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-08-31T22:49:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:49:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-08-31T22:49:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:49:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:49:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:49:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-08-31T22:49:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:49:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:49:54.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:49:54.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:49:54.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:49:54.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:49:54.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:49:54.816 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_4 2024-08-31T22:49:54.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:49:54.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:49:54.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_4 2024-08-31T22:49:54.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-08-31T22:49:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:49:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-08-31T22:49:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:49:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:49:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:49:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:49:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ purge_snapshots cluster2 mirror image_4 2024-08-31T22:49:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:49:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:49:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:49:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-08-31T22:49:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_4 2024-08-31T22:49:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:49:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:49:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:49:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:49:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:49:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_4 2024-08-31T22:49:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:49:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:49:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_4 2024-08-31T22:49:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-08-31T22:49:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-08-31T22:49:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-08-31T22:49:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:49:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:49:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_5 2024-08-31T22:49:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_5 512M 2024-08-31T22:49:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:49:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:49:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:49:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:49:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:49:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:49:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:49:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 512M ']' 2024-08-31T22:49:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=512M 2024-08-31T22:49:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:49:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_5 2024-08-31T22:49:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ enable_mirror cluster2 mirror image_5 2024-08-31T22:49:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:49:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:49:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:49:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:49:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 journal 2024-08-31T22:49:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring enabled 2024-08-31T22:49:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-08-31T22:49:55.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_5': 2024-08-31T22:49:55.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:49:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:49:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 0 2024-08-31T22:49:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 3c55b6ccc527 2024-08-31T22:49:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.3c55b6ccc527 2024-08-31T22:49:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:49:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:49:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:49:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:49:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:49:55 2024 2024-08-31T22:49:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:49:55 2024 2024-08-31T22:49:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:49:55 2024 2024-08-31T22:49:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 3c55b6ccc527 2024-08-31T22:49:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:49:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:49:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: d4ca5927-52a8-4362-b108-1a0196fa7f1a 2024-08-31T22:49:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: true 2024-08-31T22:49:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_5 2024-08-31T22:49:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:49:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:49:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:49:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:49:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=8 2024-08-31T22:49:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:49:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:49:56.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 10803200 bytes 2024-08-31T22:49:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 42698752 bytes 2024-08-31T22:50:01.320 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:300 IOs, 119337984 bytes 2024-08-31T22:50:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:50:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:50:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:50:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:50:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-08-31T22:50:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:50:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:50:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:50:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=present 2024-08-31T22:50:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:50:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:50:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:50:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_5 2024-08-31T22:50:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:50:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:50:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_5 2024-08-31T22:50:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-08-31T22:50:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:50:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id=10a854215e55 2024-08-31T22:50:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:50:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:50:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_5 present 10a854215e55 2024-08-31T22:50:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:50:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:50:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:50:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=present 2024-08-31T22:50:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id=10a854215e55 2024-08-31T22:50:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:50:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:50:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_5 2024-08-31T22:50:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:50:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:50:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_5 2024-08-31T22:50:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-08-31T22:50:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:50:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id=10a854215e55 2024-08-31T22:50:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 10a854215e55 2024-08-31T22:50:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -z 10a854215e55 -o 10a854215e55 = 10a854215e55 2024-08-31T22:50:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=present 2024-08-31T22:50:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test present = present 2024-08-31T22:50:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:50:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ disable_mirror cluster2 mirror image_5 2024-08-31T22:50:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:50:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:50:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:50:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-08-31T22:50:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring disabled 2024-08-31T22:50:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-08-31T22:50:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:50:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:50:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:50:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:50:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:50:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:50:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:50:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_5 2024-08-31T22:50:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:50:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:50:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_5 2024-08-31T22:50:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-08-31T22:50:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:50:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id=10a854215e55 2024-08-31T22:50:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:50:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:50:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_5 deleted 10a854215e55 2024-08-31T22:50:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:50:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:50:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:50:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:50:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id=10a854215e55 2024-08-31T22:50:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:50:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:50:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_5 2024-08-31T22:50:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:50:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:50:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_5 2024-08-31T22:50:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-08-31T22:50:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:50:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-08-31T22:50:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:50:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:50:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:50:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:50:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ purge_snapshots cluster2 mirror image_5 2024-08-31T22:50:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:50:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:50:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:50:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-08-31T22:50:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_5 2024-08-31T22:50:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:50:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:50:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:50:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:50:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:50:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_5 2024-08-31T22:50:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:50:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:50:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_5 2024-08-31T22:50:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-08-31T22:50:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-08-31T22:50:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-08-31T22:50:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:50:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:50:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_6 2024-08-31T22:50:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_6 512M 2024-08-31T22:50:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:50:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:50:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:50:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:50:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:50:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:50:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:50:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 512M ']' 2024-08-31T22:50:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=512M 2024-08-31T22:50:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:50:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_6 2024-08-31T22:50:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ enable_mirror cluster2 mirror image_6 2024-08-31T22:50:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:50:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:50:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:50:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:50:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 journal 2024-08-31T22:50:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring enabled 2024-08-31T22:50:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-08-31T22:50:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_6': 2024-08-31T22:50:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:50:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:50:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 0 2024-08-31T22:50:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 3c6d682ac993 2024-08-31T22:50:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.3c6d682ac993 2024-08-31T22:50:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:50:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:50:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:50:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:50:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:50:37 2024 2024-08-31T22:50:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:50:37 2024 2024-08-31T22:50:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:50:37 2024 2024-08-31T22:50:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 3c6d682ac993 2024-08-31T22:50:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:50:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:50:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 4887b332-afb4-4df1-8ac9-d8fa6a2c5f58 2024-08-31T22:50:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: true 2024-08-31T22:50:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_6 2024-08-31T22:50:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:50:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:50:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:50:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:50:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=5 2024-08-31T22:50:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:50:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:50:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 56717312 bytes 2024-08-31T22:50:42.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:50:42.303 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:50:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:50:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:50:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-08-31T22:50:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:50:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:50:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:50:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=present 2024-08-31T22:50:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:50:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:50:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:50:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_6 2024-08-31T22:50:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:50:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:50:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_6 2024-08-31T22:50:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-08-31T22:50:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:50:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id=10ab331119b3 2024-08-31T22:50:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:50:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:50:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_6 present 10ab331119b3 2024-08-31T22:50:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:50:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:50:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:50:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=present 2024-08-31T22:50:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id=10ab331119b3 2024-08-31T22:50:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:50:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:50:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_6 2024-08-31T22:50:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:50:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:50:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_6 2024-08-31T22:50:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-08-31T22:50:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:50:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id=10ab331119b3 2024-08-31T22:50:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 10ab331119b3 2024-08-31T22:50:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -z 10ab331119b3 -o 10ab331119b3 = 10ab331119b3 2024-08-31T22:50:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=present 2024-08-31T22:50:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test present = present 2024-08-31T22:50:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:50:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ disable_mirror cluster2 mirror image_6 2024-08-31T22:50:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:50:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:50:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:50:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-08-31T22:51:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring disabled 2024-08-31T22:51:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-08-31T22:51:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:51:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:51:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:51:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:51:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:51:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:51:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:51:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_6 2024-08-31T22:51:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:51:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:51:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_6 2024-08-31T22:51:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-08-31T22:51:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:51:16.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id=10ab331119b3 2024-08-31T22:51:16.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:51:16.933 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:51:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_6 deleted 10ab331119b3 2024-08-31T22:51:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:51:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:51:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:51:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:51:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id=10ab331119b3 2024-08-31T22:51:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:51:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:51:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_6 2024-08-31T22:51:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:51:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:51:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_6 2024-08-31T22:51:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-08-31T22:51:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:51:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-08-31T22:51:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:51:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:51:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:51:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:51:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ purge_snapshots cluster2 mirror image_6 2024-08-31T22:51:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:51:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:51:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:51:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-08-31T22:51:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_6 2024-08-31T22:51:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:51:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:51:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:51:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:51:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:51:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_6 2024-08-31T22:51:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:51:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:51:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_6 2024-08-31T22:51:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-08-31T22:51:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-08-31T22:51:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-08-31T22:51:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:51:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:51:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_7 2024-08-31T22:51:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_7 512M 2024-08-31T22:51:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:51:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:51:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:51:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:51:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:51:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:51:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:51:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 512M ']' 2024-08-31T22:51:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=512M 2024-08-31T22:51:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:51:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_7 2024-08-31T22:51:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ enable_mirror cluster2 mirror image_7 2024-08-31T22:51:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:51:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:51:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:51:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:51:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 journal 2024-08-31T22:51:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring enabled 2024-08-31T22:51:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-08-31T22:51:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_7': 2024-08-31T22:51:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:51:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:51:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 0 2024-08-31T22:51:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 3c854ceed277 2024-08-31T22:51:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.3c854ceed277 2024-08-31T22:51:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:51:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:51:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:51:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:51:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:51:17 2024 2024-08-31T22:51:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:51:17 2024 2024-08-31T22:51:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:51:17 2024 2024-08-31T22:51:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 3c854ceed277 2024-08-31T22:51:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:51:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:51:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 8422e179-968f-4d21-a00d-9b6e916eb35d 2024-08-31T22:51:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: true 2024-08-31T22:51:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_7 2024-08-31T22:51:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:51:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:51:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:51:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:51:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=8 2024-08-31T22:51:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:51:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:51:21.845 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 63157248 bytes 2024-08-31T22:51:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:51:25.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:51:25.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:51:25.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:51:25.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-08-31T22:51:25.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:51:25.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:51:25.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:51:25.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=present 2024-08-31T22:51:25.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:51:25.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:51:25.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:51:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_7 2024-08-31T22:51:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:51:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:51:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_7 2024-08-31T22:51:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-08-31T22:51:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:51:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id=10a82519871a 2024-08-31T22:51:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:51:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:51:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_7 present 10a82519871a 2024-08-31T22:51:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:51:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:51:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:51:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=present 2024-08-31T22:51:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id=10a82519871a 2024-08-31T22:51:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:51:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:51:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_7 2024-08-31T22:51:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:51:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:51:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_7 2024-08-31T22:51:25.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-08-31T22:51:25.739 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:51:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id=10a82519871a 2024-08-31T22:51:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 10a82519871a 2024-08-31T22:51:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -z 10a82519871a -o 10a82519871a = 10a82519871a 2024-08-31T22:51:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=present 2024-08-31T22:51:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test present = present 2024-08-31T22:51:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:51:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ disable_mirror cluster2 mirror image_7 2024-08-31T22:51:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:51:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:51:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:51:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-08-31T22:51:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring disabled 2024-08-31T22:51:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-08-31T22:51:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:51:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:51:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:51:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:51:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:51:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:51:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:51:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_7 2024-08-31T22:51:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:51:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:51:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_7 2024-08-31T22:51:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-08-31T22:51:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:51:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-08-31T22:51:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:51:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:51:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:51:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-08-31T22:51:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:51:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:51:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:51:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:51:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:51:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:51:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:51:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_7 2024-08-31T22:51:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:51:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:51:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_7 2024-08-31T22:51:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-08-31T22:51:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:51:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-08-31T22:51:53.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:51:53.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:51:53.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:51:53.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:51:53.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ purge_snapshots cluster2 mirror image_7 2024-08-31T22:51:53.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:51:53.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:51:53.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:51:53.804 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-08-31T22:51:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_7 2024-08-31T22:51:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:51:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:51:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:51:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:51:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:51:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_7 2024-08-31T22:51:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:51:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:51:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_7 2024-08-31T22:51:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-08-31T22:51:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-08-31T22:51:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-08-31T22:51:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:51:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:51:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_8 2024-08-31T22:51:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_8 512M 2024-08-31T22:51:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:51:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:51:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:51:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:51:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:51:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:51:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:51:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 512M ']' 2024-08-31T22:51:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=512M 2024-08-31T22:51:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:51:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_8 2024-08-31T22:51:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ enable_mirror cluster2 mirror image_8 2024-08-31T22:51:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:51:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:51:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:51:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:51:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 journal 2024-08-31T22:51:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring enabled 2024-08-31T22:51:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-08-31T22:51:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_8': 2024-08-31T22:51:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:51:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:51:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 0 2024-08-31T22:51:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 3c9dbb6c4edb 2024-08-31T22:51:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.3c9dbb6c4edb 2024-08-31T22:51:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:51:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:51:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:51:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:51:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:51:54 2024 2024-08-31T22:51:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:51:54 2024 2024-08-31T22:51:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:51:54 2024 2024-08-31T22:51:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 3c9dbb6c4edb 2024-08-31T22:51:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:51:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:51:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: b1cacd3a-6d38-4b96-9aa6-339658e78cc6 2024-08-31T22:51:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: true 2024-08-31T22:51:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_8 2024-08-31T22:51:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:51:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:51:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:51:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:51:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=11 2024-08-31T22:51:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:51:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:52:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 144587776 bytes 2024-08-31T22:52:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:52:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:52:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:52:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:52:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-08-31T22:52:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:52:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:52:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:52:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=present 2024-08-31T22:52:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:52:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:52:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:52:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_8 2024-08-31T22:52:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:52:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:52:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_8 2024-08-31T22:52:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-08-31T22:52:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:52:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id=10a894ef9e0b 2024-08-31T22:52:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:52:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:52:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_8 present 10a894ef9e0b 2024-08-31T22:52:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:52:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:52:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:52:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=present 2024-08-31T22:52:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id=10a894ef9e0b 2024-08-31T22:52:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:52:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:52:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_8 2024-08-31T22:52:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:52:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:52:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_8 2024-08-31T22:52:05.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-08-31T22:52:05.497 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:52:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id=10a894ef9e0b 2024-08-31T22:52:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 10a894ef9e0b 2024-08-31T22:52:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -z 10a894ef9e0b -o 10a894ef9e0b = 10a894ef9e0b 2024-08-31T22:52:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=present 2024-08-31T22:52:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test present = present 2024-08-31T22:52:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:52:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ disable_mirror cluster2 mirror image_8 2024-08-31T22:52:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:52:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:52:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:52:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-08-31T22:52:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring disabled 2024-08-31T22:52:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-08-31T22:52:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:52:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:52:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:52:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:52:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:52:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:52:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:52:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_8 2024-08-31T22:52:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:52:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:52:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_8 2024-08-31T22:52:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-08-31T22:52:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:52:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id=10a894ef9e0b 2024-08-31T22:52:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:52:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:52:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_8 deleted 10a894ef9e0b 2024-08-31T22:52:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:52:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:52:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:52:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:52:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id=10a894ef9e0b 2024-08-31T22:52:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:52:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:52:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_8 2024-08-31T22:52:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:52:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:52:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_8 2024-08-31T22:52:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-08-31T22:52:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:52:45.248 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-08-31T22:52:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:52:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:52:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:52:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:52:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ purge_snapshots cluster2 mirror image_8 2024-08-31T22:52:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:52:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:52:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:52:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-08-31T22:52:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_8 2024-08-31T22:52:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:52:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:52:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:52:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:52:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:52:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_8 2024-08-31T22:52:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:52:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:52:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_8 2024-08-31T22:52:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-08-31T22:52:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-08-31T22:52:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-08-31T22:52:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:52:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:52:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_9 2024-08-31T22:52:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_9 512M 2024-08-31T22:52:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:52:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:52:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:52:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:52:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:52:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:52:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:52:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 512M ']' 2024-08-31T22:52:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=512M 2024-08-31T22:52:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:52:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_9 2024-08-31T22:52:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ enable_mirror cluster2 mirror image_9 2024-08-31T22:52:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:52:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:52:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:52:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:52:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 journal 2024-08-31T22:52:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring enabled 2024-08-31T22:52:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-08-31T22:52:45.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_9': 2024-08-31T22:52:45.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:52:45.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:52:45.750 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 0 2024-08-31T22:52:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 3cb51a17a58f 2024-08-31T22:52:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.3cb51a17a58f 2024-08-31T22:52:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:52:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:52:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:52:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:52:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:52:45 2024 2024-08-31T22:52:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:52:45 2024 2024-08-31T22:52:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:52:45 2024 2024-08-31T22:52:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 3cb51a17a58f 2024-08-31T22:52:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:52:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:52:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 938d291e-353c-4461-84a4-625e6f695793 2024-08-31T22:52:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: true 2024-08-31T22:52:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_9 2024-08-31T22:52:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:52:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:52:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:52:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:52:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=6 2024-08-31T22:52:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:52:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:52:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 48941056 bytes 2024-08-31T22:52:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:52:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:52:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:52:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:52:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-08-31T22:52:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:52:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:52:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:52:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=present 2024-08-31T22:52:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:52:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:52:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:52:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_9 2024-08-31T22:52:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:52:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:52:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_9 2024-08-31T22:52:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-08-31T22:52:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:52:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id=10ab6f9dfc2f 2024-08-31T22:52:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:52:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:52:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_9 present 10ab6f9dfc2f 2024-08-31T22:52:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:52:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:52:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:52:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=present 2024-08-31T22:52:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id=10ab6f9dfc2f 2024-08-31T22:52:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:52:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:52:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_9 2024-08-31T22:52:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:52:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:52:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_9 2024-08-31T22:52:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-08-31T22:52:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:52:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id=10ab6f9dfc2f 2024-08-31T22:52:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 10ab6f9dfc2f 2024-08-31T22:52:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -z 10ab6f9dfc2f -o 10ab6f9dfc2f = 10ab6f9dfc2f 2024-08-31T22:52:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=present 2024-08-31T22:52:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test present = present 2024-08-31T22:52:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:52:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ disable_mirror cluster2 mirror image_9 2024-08-31T22:52:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:52:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:52:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:52:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-08-31T22:53:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring disabled 2024-08-31T22:53:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-08-31T22:53:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:53:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:53:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:53:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:53:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:53:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:53:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:53:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_9 2024-08-31T22:53:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:53:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:53:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_9 2024-08-31T22:53:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:53:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-08-31T22:53:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-08-31T22:53:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:53:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:53:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:53:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-08-31T22:53:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:53:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:53:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:53:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:53:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:53:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:53:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:53:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_9 2024-08-31T22:53:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:53:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:53:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_9 2024-08-31T22:53:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-08-31T22:53:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:53:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-08-31T22:53:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:53:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:53:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:53:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:53:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ purge_snapshots cluster2 mirror image_9 2024-08-31T22:53:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:53:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:53:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:53:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-08-31T22:53:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_9 2024-08-31T22:53:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:53:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:53:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:53:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:53:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:53:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_9 2024-08-31T22:53:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:53:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:53:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_9 2024-08-31T22:53:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-08-31T22:53:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-08-31T22:53:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-08-31T22:53:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:53:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for i in `seq 1 10` 2024-08-31T22:53:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image=image_10 2024-08-31T22:53:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ create_image cluster2 mirror image_10 512M 2024-08-31T22:53:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:53:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:53:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:53:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:53:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:53:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:53:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local size=128 2024-08-31T22:53:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 512M ']' 2024-08-31T22:53:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ size=512M 2024-08-31T22:53:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ shift 2024-08-31T22:53:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_10 2024-08-31T22:53:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ enable_mirror cluster2 mirror image_10 2024-08-31T22:53:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:53:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:53:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:53:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local mode=journal 2024-08-31T22:53:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 journal 2024-08-31T22:53:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring enabled 2024-08-31T22:53:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-08-31T22:53:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:rbd image 'image_10': 2024-08-31T22:53:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: size 512 MiB in 128 objects 2024-08-31T22:53:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: order 22 (4 MiB objects) 2024-08-31T22:53:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: snapshot_count: 0 2024-08-31T22:53:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: id: 3ccddc61fb4b 2024-08-31T22:53:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: block_name_prefix: rbd_data.3ccddc61fb4b 2024-08-31T22:53:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: format: 2 2024-08-31T22:53:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: features: layering, exclusive-lock, journaling 2024-08-31T22:53:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: op_features: 2024-08-31T22:53:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: flags: 2024-08-31T22:53:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: create_timestamp: Sat Aug 31 22:53:30 2024 2024-08-31T22:53:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: access_timestamp: Sat Aug 31 22:53:30 2024 2024-08-31T22:53:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: modify_timestamp: Sat Aug 31 22:53:30 2024 2024-08-31T22:53:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: journal: 3ccddc61fb4b 2024-08-31T22:53:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring state: enabled 2024-08-31T22:53:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring mode: journal 2024-08-31T22:53:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring global id: 57b18ade-18df-4210-aecf-0368541ba0b4 2024-08-31T22:53:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout: mirroring primary: true 2024-08-31T22:53:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stress_write_image cluster2 mirror image_10 2024-08-31T22:53:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:53:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:53:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:53:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-08-31T22:53:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local duration=14 2024-08-31T22:53:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:53:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-08-31T22:53:37.375 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:53:37.372+0000 7f13efcb9640 -1 reset not still connected to 0x559f0fd4c680 2024-08-31T22:53:37.375 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:53:37.372+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff161a0 2024-08-31T22:53:37.375 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:53:37.372+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff16d00 2024-08-31T22:53:37.376 INFO:tasks.ceph.cluster2.osd.2.smithi032.stderr:2024-08-31T22:53:37.372+0000 7f13efcb9640 -1 reset not still connected to 0x559f0ff17c70 2024-08-31T22:53:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:100 IOs, 122800128 bytes 2024-08-31T22:53:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:200 IOs, 180757504 bytes 2024-08-31T22:53:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=124 2024-08-31T22:53:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:53:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 124 -eq 124 ']' 2024-08-31T22:53:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:53:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-08-31T22:53:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:53:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:53:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:53:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=present 2024-08-31T22:53:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:53:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:53:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:53:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_10 2024-08-31T22:53:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:53:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:53:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_10 2024-08-31T22:53:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-08-31T22:53:44.397 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:53:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id=10a89824c1d6 2024-08-31T22:53:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:53:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:53:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_10 present 10a89824c1d6 2024-08-31T22:53:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:53:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:53:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:53:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=present 2024-08-31T22:53:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id=10a89824c1d6 2024-08-31T22:53:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:53:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:53:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_10 2024-08-31T22:53:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:53:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:53:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_10 2024-08-31T22:53:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-08-31T22:53:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:53:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id=10a89824c1d6 2024-08-31T22:53:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n 10a89824c1d6 2024-08-31T22:53:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -z 10a89824c1d6 -o 10a89824c1d6 = 10a89824c1d6 2024-08-31T22:53:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_state=present 2024-08-31T22:53:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test present = present 2024-08-31T22:53:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:53:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ disable_mirror cluster2 mirror image_10 2024-08-31T22:53:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:53:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:53:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:53:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-08-31T22:54:14.737 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:Mirroring disabled 2024-08-31T22:54:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-08-31T22:54:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:54:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:54:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:54:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local state=deleted 2024-08-31T22:54:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:54:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local s 2024-08-31T22:54:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:54:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_10 2024-08-31T22:54:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:54:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:54:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_10 2024-08-31T22:54:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-08-31T22:54:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:54:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-08-31T22:54:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ image_id= 2024-08-31T22:54:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-08-31T22:54:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0.1 2024-08-31T22:54:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-08-31T22:54:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:54:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:54:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:54:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local test_state=deleted 2024-08-31T22:54:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image_id= 2024-08-31T22:54:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_state=deleted 2024-08-31T22:54:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local current_image_id 2024-08-31T22:54:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ get_image_id cluster1 mirror image_10 2024-08-31T22:54:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:54:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:54:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local image=image_10 2024-08-31T22:54:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-08-31T22:54:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-08-31T22:54:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-08-31T22:54:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ current_image_id= 2024-08-31T22:54:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:54:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test deleted = deleted 2024-08-31T22:54:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:54:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ purge_snapshots cluster2 mirror image_10 2024-08-31T22:54:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:54:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:54:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:54:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-08-31T22:54:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image_retry cluster2 mirror image_10 2024-08-31T22:54:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:54:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:54:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:54:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 4 8 16 32 2024-08-31T22:54:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:54:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ remove_image cluster2 mirror image_10 2024-08-31T22:54:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:54:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:54:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local image=image_10 2024-08-31T22:54:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-08-31T22:54:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-08-31T22:54:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-08-31T22:54:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:54:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-08-31T22:54:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ tee -a /tmp/tmp.iYfzsjLopX/rbd-mirror.test.log 2024-08-31T22:54:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ date '+%F %T' 2024-08-31T22:54:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo 2024-08-31 22:54:15 TEST: check if removed 'images'\''' OMAP are removed 2024-08-31T22:54:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:2024-08-31 22:54:15 TEST: check if removed images' OMAP are removed 2024-08-31T22:54:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-08-31T22:54:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:54:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:54:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-08-31T22:54:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:54:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:54:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local obj_name=rbd_mirroring 2024-08-31T22:54:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local filter=status_global 2024-08-31T22:54:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-08-31T22:54:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:54:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:54:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-08-31T22:54:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:54:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:54:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local obj_name=rbd_mirroring 2024-08-31T22:54:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local filter=status_global 2024-08-31T22:54:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-08-31T22:54:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:54:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:54:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local obj_name=rbd_mirroring 2024-08-31T22:54:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -c status_global 2024-08-31T22:54:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-08-31T22:54:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 0 = 0 2024-08-31T22:54:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=0 2024-08-31T22:54:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:54:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 -eq 0 ']' 2024-08-31T22:54:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:54:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-08-31T22:54:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:54:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:54:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local obj_name=rbd_mirroring 2024-08-31T22:54:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local filter=image_ 2024-08-31T22:54:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-08-31T22:54:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:54:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:54:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-08-31T22:54:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:54:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:54:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local obj_name=rbd_mirroring 2024-08-31T22:54:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local filter=image_ 2024-08-31T22:54:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-08-31T22:54:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:54:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -c image_ 2024-08-31T22:54:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:54:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local obj_name=rbd_mirroring 2024-08-31T22:54:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-08-31T22:54:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 0 = 0 2024-08-31T22:54:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=0 2024-08-31T22:54:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:54:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 -eq 0 ']' 2024-08-31T22:54:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:54:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-08-31T22:54:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:54:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:54:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local obj_name=rbd_mirror_leader 2024-08-31T22:54:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local filter=image_map 2024-08-31T22:54:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-08-31T22:54:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:54:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:54:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-08-31T22:54:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:54:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:54:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local obj_name=rbd_mirror_leader 2024-08-31T22:54:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local filter=image_map 2024-08-31T22:54:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-08-31T22:54:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:54:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:54:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -c image_map 2024-08-31T22:54:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local obj_name=rbd_mirror_leader 2024-08-31T22:54:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-08-31T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 0 = 0 2024-08-31T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=0 2024-08-31T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 -eq 0 ']' 2024-08-31T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-08-31T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:54:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-08-31T22:54:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:54:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:54:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local obj_name=rbd_mirroring 2024-08-31T22:54:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local filter=status_global 2024-08-31T22:54:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-08-31T22:54:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:54:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:54:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-08-31T22:54:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:54:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:54:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local obj_name=rbd_mirroring 2024-08-31T22:54:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local filter=status_global 2024-08-31T22:54:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-08-31T22:54:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:54:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:54:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local obj_name=rbd_mirroring 2024-08-31T22:54:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -c status_global 2024-08-31T22:54:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-08-31T22:54:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 0 = 0 2024-08-31T22:54:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=0 2024-08-31T22:54:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:54:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 -eq 0 ']' 2024-08-31T22:54:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:54:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-08-31T22:54:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:54:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:54:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local obj_name=rbd_mirroring 2024-08-31T22:54:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local filter=image_ 2024-08-31T22:54:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-08-31T22:54:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:54:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:54:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-08-31T22:54:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:54:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:54:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local obj_name=rbd_mirroring 2024-08-31T22:54:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local filter=image_ 2024-08-31T22:54:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-08-31T22:54:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:54:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -c image_ 2024-08-31T22:54:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:54:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local obj_name=rbd_mirroring 2024-08-31T22:54:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-08-31T22:54:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 0 = 0 2024-08-31T22:54:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=0 2024-08-31T22:54:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:54:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 -eq 0 ']' 2024-08-31T22:54:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:54:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-08-31T22:54:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:54:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pool=mirror 2024-08-31T22:54:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local obj_name=rbd_mirror_leader 2024-08-31T22:54:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local filter=image_map 2024-08-31T22:54:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-08-31T22:54:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 0 2024-08-31T22:54:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:54:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-08-31T22:54:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:54:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:54:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local obj_name=rbd_mirror_leader 2024-08-31T22:54:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local filter=image_map 2024-08-31T22:54:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-08-31T22:54:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:54:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local pool=mirror 2024-08-31T22:54:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local obj_name=rbd_mirror_leader 2024-08-31T22:54:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-08-31T22:54:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ grep -c image_map 2024-08-31T22:54:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test 0 = 0 2024-08-31T22:54:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ error_code=0 2024-08-31T22:54:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set -e 2024-08-31T22:54:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 -eq 0 ']' 2024-08-31T22:54:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ return 0 2024-08-31T22:54:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ cleanup 0 2024-08-31T22:54:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local error_code=0 2024-08-31T22:54:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ set +e 2024-08-31T22:54:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 -ne 0 ']' 2024-08-31T22:54:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -z '' ']' 2024-08-31T22:54:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster instance 2024-08-31T22:54:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ CEPH_ARGS= 2024-08-31T22:54:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-08-31T22:54:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:pool 'mirror' does not exist 2024-08-31T22:54:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ CEPH_ARGS= 2024-08-31T22:54:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-08-31T22:54:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:pool 'mirror' does not exist 2024-08-31T22:54:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ CEPH_ARGS= 2024-08-31T22:54:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-08-31T22:54:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:pool 'mirror_parent' does not exist 2024-08-31T22:54:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ CEPH_ARGS= 2024-08-31T22:54:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-08-31T22:54:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:pool 'mirror_parent' does not exist 2024-08-31T22:54:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-08-31T22:54:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stop_mirrors cluster1 2024-08-31T22:54:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1 2024-08-31T22:54:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local sig= 2024-08-31T22:54:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:54:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:54:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stop_mirror cluster1:0 '' 2024-08-31T22:54:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:0 2024-08-31T22:54:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local sig= 2024-08-31T22:54:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:54:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pid 2024-08-31T22:54:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_pid_file cluster1:0 2024-08-31T22:54:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1:0 2024-08-31T22:54:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:54:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:54:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:0 2024-08-31T22:54:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=cluster 2024-08-31T22:54:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:54:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:54:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:54:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster1:0 ']' 2024-08-31T22:54:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval cluster=cluster1 2024-08-31T22:54:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ cluster=cluster1 2024-08-31T22:54:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:54:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:54:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-08-31T22:54:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-08-31T22:54:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-08-31T22:54:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ pid=29512 2024-08-31T22:54:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 29512 ']' 2024-08-31T22:54:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ kill 29512 2024-08-31T22:54:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 16 32 2024-08-31T22:54:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:54:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ps auxww 2024-08-31T22:54:20.692 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ awk -v pid=29512 '$2 == pid {print; exit 1}' 2024-08-31T22:54:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:54:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ps auxww 2024-08-31T22:54:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ awk -v pid=29512 '$2 == pid {print; exit 1}' 2024-08-31T22:54:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-08-31T22:54:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:54:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:54:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:54:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:54:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:54:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:54:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:54:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:54:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:54:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:54:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:54:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:54:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:54:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:54:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:54:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:54:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:54:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:0 cluster2 2024-08-31T22:54:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:0 2024-08-31T22:54:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:54:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:54:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-08-31T22:54:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:54:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:54:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:54:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:54:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:54:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:54:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:54:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:54:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:54:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:54:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-08-31T22:54:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:54:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-08-31T22:54:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_pid_file cluster1:0 2024-08-31T22:54:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:0 2024-08-31T22:54:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:54:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-08-31T22:54:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:0 2024-08-31T22:54:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:54:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:54:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:54:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:54:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster1:0 ']' 2024-08-31T22:54:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:54:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:54:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:54:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:54:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-08-31T22:54:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-08-31T22:54:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-08-31T22:54:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:54:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stop_mirror cluster1:1 '' 2024-08-31T22:54:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster1:1 2024-08-31T22:54:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local sig= 2024-08-31T22:54:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:54:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pid 2024-08-31T22:54:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_pid_file cluster1:1 2024-08-31T22:54:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster1:1 2024-08-31T22:54:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:54:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:54:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster1:1 2024-08-31T22:54:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=cluster 2024-08-31T22:54:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:54:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:54:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:54:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster1:1 ']' 2024-08-31T22:54:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval cluster=cluster1 2024-08-31T22:54:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ cluster=cluster1 2024-08-31T22:54:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:54:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:54:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-08-31T22:54:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-08-31T22:54:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-08-31T22:54:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ pid=29540 2024-08-31T22:54:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 29540 ']' 2024-08-31T22:54:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ kill 29540 2024-08-31T22:54:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 16 32 2024-08-31T22:54:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:54:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ps auxww 2024-08-31T22:54:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ awk -v pid=29540 '$2 == pid {print; exit 1}' 2024-08-31T22:54:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:54:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ps auxww 2024-08-31T22:54:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ awk -v pid=29540 '$2 == pid {print; exit 1}' 2024-08-31T22:54:21.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-08-31T22:54:21.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:54:21.909 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:54:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:54:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:54:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:54:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:54:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:54:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:54:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:54:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:54:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:54:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:54:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:54:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:54:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:54:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:54:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:54:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster1:1 cluster2 2024-08-31T22:54:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster1:1 2024-08-31T22:54:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:54:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:54:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-08-31T22:54:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:54:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:54:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:54:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:54:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:54:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:54:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster1 2024-08-31T22:54:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster1 2024-08-31T22:54:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:54:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:54:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-08-31T22:54:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:54:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-08-31T22:54:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_pid_file cluster1:1 2024-08-31T22:54:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1:1 2024-08-31T22:54:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:54:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-08-31T22:54:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster1:1 2024-08-31T22:54:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:54:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:54:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster1 2024-08-31T22:54:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:54:21.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster1:1 ']' 2024-08-31T22:54:21.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster1 2024-08-31T22:54:21.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster1 2024-08-31T22:54:21.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:54:21.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:54:21.968 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-08-31T22:54:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-08-31T22:54:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-08-31T22:54:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-08-31T22:54:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stop_mirrors cluster2 2024-08-31T22:54:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2 2024-08-31T22:54:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local sig= 2024-08-31T22:54:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ seq 0 1 2024-08-31T22:54:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:54:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stop_mirror cluster2:0 '' 2024-08-31T22:54:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2:0 2024-08-31T22:54:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local sig= 2024-08-31T22:54:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:54:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pid 2024-08-31T22:54:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_pid_file cluster2:0 2024-08-31T22:54:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster2:0 2024-08-31T22:54:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:54:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster2:0 cluster instance 2024-08-31T22:54:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster2:0 2024-08-31T22:54:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=cluster 2024-08-31T22:54:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:54:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster2 2024-08-31T22:54:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:54:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 0 = cluster2:0 ']' 2024-08-31T22:54:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval cluster=cluster2 2024-08-31T22:54:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ cluster=cluster2 2024-08-31T22:54:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=0 2024-08-31T22:54:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=0 2024-08-31T22:54:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-08-31T22:54:22.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-08-31T22:54:22.022 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-08-31T22:54:22.023 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ pid=29577 2024-08-31T22:54:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 29577 ']' 2024-08-31T22:54:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ kill 29577 2024-08-31T22:54:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 16 32 2024-08-31T22:54:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:54:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ps auxww 2024-08-31T22:54:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ awk -v pid=29577 '$2 == pid {print; exit 1}' 2024-08-31T22:54:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:54:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ps auxww 2024-08-31T22:54:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ awk -v pid=29577 '$2 == pid {print; exit 1}' 2024-08-31T22:54:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster2:0 cluster1 2024-08-31T22:54:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster2:0 2024-08-31T22:54:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:54:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:54:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-08-31T22:54:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster2:0 2024-08-31T22:54:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:54:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:54:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster2 2024-08-31T22:54:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:54:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster2:0 ']' 2024-08-31T22:54:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster2 2024-08-31T22:54:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster2 2024-08-31T22:54:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:54:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:54:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-08-31T22:54:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-08-31T22:54:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-08-31T22:54:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-08-31T22:54:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster2:0 2024-08-31T22:54:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:54:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:54:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-08-31T22:54:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster2:0 2024-08-31T22:54:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:54:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:54:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster2 2024-08-31T22:54:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:54:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster2:0 ']' 2024-08-31T22:54:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster2 2024-08-31T22:54:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster2 2024-08-31T22:54:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:54:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:54:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-08-31T22:54:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-08-31T22:54:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-08-31T22:54:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_pid_file cluster2:0 2024-08-31T22:54:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2:0 2024-08-31T22:54:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:54:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-08-31T22:54:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster2:0 2024-08-31T22:54:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:54:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:54:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster2 2024-08-31T22:54:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=0 2024-08-31T22:54:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 0 = cluster2:0 ']' 2024-08-31T22:54:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster2 2024-08-31T22:54:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster2 2024-08-31T22:54:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=0 2024-08-31T22:54:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=0 2024-08-31T22:54:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-08-31T22:54:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-08-31T22:54:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-08-31T22:54:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-08-31T22:54:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ stop_mirror cluster2:1 '' 2024-08-31T22:54:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local cluster=cluster2:1 2024-08-31T22:54:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local sig= 2024-08-31T22:54:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test -n '' 2024-08-31T22:54:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ local pid 2024-08-31T22:54:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ daemon_pid_file cluster2:1 2024-08-31T22:54:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster=cluster2:1 2024-08-31T22:54:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance 2024-08-31T22:54:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ set_cluster_instance cluster2:1 cluster instance 2024-08-31T22:54:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local val=cluster2:1 2024-08-31T22:54:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local cluster_var_name=cluster 2024-08-31T22:54:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local instance_var_name=instance 2024-08-31T22:54:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster2 2024-08-31T22:54:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:54:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ '[' 1 = cluster2:1 ']' 2024-08-31T22:54:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval cluster=cluster2 2024-08-31T22:54:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ cluster=cluster2 2024-08-31T22:54:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ eval instance=1 2024-08-31T22:54:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ instance=1 2024-08-31T22:54:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-08-31T22:54:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-08-31T22:54:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-08-31T22:54:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ pid=29645 2024-08-31T22:54:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -n 29645 ']' 2024-08-31T22:54:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ kill 29645 2024-08-31T22:54:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ for s in 1 2 4 8 16 32 2024-08-31T22:54:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ sleep 1 2024-08-31T22:54:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ps auxww 2024-08-31T22:54:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ awk -v pid=29645 '$2 == pid {print; exit 1}' 2024-08-31T22:54:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ break 2024-08-31T22:54:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ ps auxww 2024-08-31T22:54:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ awk -v pid=29645 '$2 == pid {print; exit 1}' 2024-08-31T22:54:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster2:1 cluster1 2024-08-31T22:54:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster2:1 2024-08-31T22:54:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster1 2024-08-31T22:54:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:54:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-08-31T22:54:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster2:1 2024-08-31T22:54:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:54:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:54:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster2 2024-08-31T22:54:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:54:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster2:1 ']' 2024-08-31T22:54:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster2 2024-08-31T22:54:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster2 2024-08-31T22:54:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:54:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:54:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-08-31T22:54:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-08-31T22:54:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-08-31T22:54:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-08-31T22:54:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local local_cluster=cluster2:1 2024-08-31T22:54:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2 2024-08-31T22:54:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:54:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-08-31T22:54:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster2:1 2024-08-31T22:54:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=local_cluster 2024-08-31T22:54:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:54:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster2 2024-08-31T22:54:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:54:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster2:1 ']' 2024-08-31T22:54:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval local_cluster=cluster2 2024-08-31T22:54:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ local_cluster=cluster2 2024-08-31T22:54:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:54:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:54:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-08-31T22:54:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-08-31T22:54:24.257 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-08-31T22:54:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ daemon_pid_file cluster2:1 2024-08-31T22:54:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster=cluster2:1 2024-08-31T22:54:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance 2024-08-31T22:54:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ set_cluster_instance cluster2:1 cluster instance 2024-08-31T22:54:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local val=cluster2:1 2024-08-31T22:54:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local cluster_var_name=cluster 2024-08-31T22:54:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ local instance_var_name=instance 2024-08-31T22:54:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ cluster=cluster2 2024-08-31T22:54:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ instance=1 2024-08-31T22:54:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ '[' 1 = cluster2:1 ']' 2024-08-31T22:54:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval cluster=cluster2 2024-08-31T22:54:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ cluster=cluster2 2024-08-31T22:54:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ eval instance=1 2024-08-31T22:54:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ instance=1 2024-08-31T22:54:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-08-31T22:54:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-08-31T22:54:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-08-31T22:54:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' -z 1 ']' 2024-08-31T22:54:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ test '' = /tmp/tmp.iYfzsjLopX 2024-08-31T22:54:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ rm -Rf /tmp/tmp.iYfzsjLopX 2024-08-31T22:54:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ '[' 0 -eq 0 ']' 2024-08-31T22:54:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ echo OK 2024-08-31T22:54:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stdout:OK 2024-08-31T22:54:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi032.stderr:+ exit 0 2024-08-31T22:54:24.300 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-31T22:54:24.301 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-08-31T22:54:24.491 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-08-31T22:54:24.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-08-31T22:54:24.777 DEBUG:teuthology.parallel:result is None 2024-08-31T22:54:24.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-08-31T22:54:24.789 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-08-31T22:54:24.789 DEBUG:teuthology.orchestra.run.smithi032:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-08-31T22:54:24.836 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-08-31T22:54:24.836 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-08-31T22:54:24.845 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-08-31T22:54:24.846 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-08-31T22:54:25.186 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:54:25.187 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-31T22:54:25.205 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1640,"stamp":"2024-08-31T22:54:23.394779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":227,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3109016,"kb_used_data":14304,"kb_used_omap":358,"kb_used_meta":3094297,"kb_avail":280006504,"statfs":{"total":289910292480,"available":286726660096,"internally_reserved":0,"allocated":14647296,"data_stored":8095107,"data_compressed":6389,"data_compressed_allocated":77824,"data_compressed_original":1155072,"omap_allocated":366742,"internal_metadata":3168561002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-469708854,"num_objects":-161,"num_object_clones":-3,"num_object_copies":-322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-161,"num_whiteouts":-3,"num_read":-991900,"num_read_kb":-46245730,"num_write":-1199108,"num_write_kb":-23997177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-44315535,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002569"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":823,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.551816+0000","last_change":"2024-08-31T22:13:27.638168+0000","last_active":"2024-08-31T22:54:19.551816+0000","last_peered":"2024-08-31T22:54:19.551816+0000","last_clean":"2024-08-31T22:54:19.551816+0000","last_became_active":"2024-08-31T22:13:25.634194+0000","last_became_peered":"2024-08-31T22:13:25.634194+0000","last_unstale":"2024-08-31T22:54:19.551816+0000","last_undegraded":"2024-08-31T22:54:19.551816+0000","last_fullsized":"2024-08-31T22:54:19.551816+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T03:43:12.617984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024029400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":823,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.547368+0000","last_change":"2024-08-31T22:13:27.636754+0000","last_active":"2024-08-31T22:54:19.547368+0000","last_peered":"2024-08-31T22:54:19.547368+0000","last_clean":"2024-08-31T22:54:19.547368+0000","last_became_active":"2024-08-31T22:13:25.634366+0000","last_became_peered":"2024-08-31T22:13:25.634366+0000","last_unstale":"2024-08-31T22:54:19.547368+0000","last_undegraded":"2024-08-31T22:54:19.547368+0000","last_fullsized":"2024-08-31T22:54:19.547368+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:32:00.532608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001106084,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.559966+0000","last_change":"2024-08-31T22:13:27.637651+0000","last_active":"2024-08-31T22:54:19.559966+0000","last_peered":"2024-08-31T22:54:19.559966+0000","last_clean":"2024-08-31T22:54:19.559966+0000","last_became_active":"2024-08-31T22:13:25.634100+0000","last_became_peered":"2024-08-31T22:13:25.634100+0000","last_unstale":"2024-08-31T22:54:19.559966+0000","last_undegraded":"2024-08-31T22:54:19.559966+0000","last_fullsized":"2024-08-31T22:54:19.559966+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T09:42:20.559113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001870128,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.514353+0000","last_change":"2024-08-31T22:13:27.637511+0000","last_active":"2024-08-31T22:54:19.514353+0000","last_peered":"2024-08-31T22:54:19.514353+0000","last_clean":"2024-08-31T22:54:19.514353+0000","last_became_active":"2024-08-31T22:13:25.628362+0000","last_became_peered":"2024-08-31T22:13:25.628362+0000","last_unstale":"2024-08-31T22:54:19.514353+0000","last_undegraded":"2024-08-31T22:54:19.514353+0000","last_fullsized":"2024-08-31T22:54:19.514353+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T07:08:57.946059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001773806,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":824,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-08-31T22:54:18.444433+0000","last_change":"2024-08-31T22:13:27.640074+0000","last_active":"2024-08-31T22:54:18.444433+0000","last_peered":"2024-08-31T22:54:18.444433+0000","last_clean":"2024-08-31T22:54:18.444433+0000","last_became_active":"2024-08-31T22:13:25.628711+0000","last_became_peered":"2024-08-31T22:13:25.628711+0000","last_unstale":"2024-08-31T22:54:18.444433+0000","last_undegraded":"2024-08-31T22:54:18.444433+0000","last_fullsized":"2024-08-31T22:54:18.444433+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T08:42:16.373330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002200956,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":820,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-08-31T22:54:17.498849+0000","last_change":"2024-08-31T22:13:27.762539+0000","last_active":"2024-08-31T22:54:17.498849+0000","last_peered":"2024-08-31T22:54:17.498849+0000","last_clean":"2024-08-31T22:54:17.498849+0000","last_became_active":"2024-08-31T22:13:25.627201+0000","last_became_peered":"2024-08-31T22:13:25.627201+0000","last_unstale":"2024-08-31T22:54:17.498849+0000","last_undegraded":"2024-08-31T22:54:17.498849+0000","last_fullsized":"2024-08-31T22:54:17.498849+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T09:46:26.013807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054468599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":821,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-08-31T22:54:17.504974+0000","last_change":"2024-08-31T22:13:27.762844+0000","last_active":"2024-08-31T22:54:17.504974+0000","last_peered":"2024-08-31T22:54:17.504974+0000","last_clean":"2024-08-31T22:54:17.504974+0000","last_became_active":"2024-08-31T22:13:25.631027+0000","last_became_peered":"2024-08-31T22:13:25.631027+0000","last_unstale":"2024-08-31T22:54:17.504974+0000","last_undegraded":"2024-08-31T22:54:17.504974+0000","last_fullsized":"2024-08-31T22:54:17.504974+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:16:59.947661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060313200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":825,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.541603+0000","last_change":"2024-08-31T22:13:27.636882+0000","last_active":"2024-08-31T22:54:19.541603+0000","last_peered":"2024-08-31T22:54:19.541603+0000","last_clean":"2024-08-31T22:54:19.541603+0000","last_became_active":"2024-08-31T22:13:25.628867+0000","last_became_peered":"2024-08-31T22:13:25.628867+0000","last_unstale":"2024-08-31T22:54:19.541603+0000","last_undegraded":"2024-08-31T22:54:19.541603+0000","last_fullsized":"2024-08-31T22:54:19.541603+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:00:38.825010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001191278,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":953,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.515864+0000","last_change":"2024-08-31T22:13:21.644125+0000","last_active":"2024-08-31T22:54:19.515864+0000","last_peered":"2024-08-31T22:54:19.515864+0000","last_clean":"2024-08-31T22:54:19.515864+0000","last_became_active":"2024-08-31T22:13:21.641333+0000","last_became_peered":"2024-08-31T22:13:21.641333+0000","last_unstale":"2024-08-31T22:54:19.515864+0000","last_undegraded":"2024-08-31T22:54:19.515864+0000","last_fullsized":"2024-08-31T22:54:19.515864+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:20.589879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:20.589879+0000","last_clean_scrub_stamp":"2024-08-31T22:13:20.589879+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:38:29.946497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6354,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706164,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607616,"kb_used_data":4720,"kb_used_omap":123,"kb_used_meta":602756,"kb_avail":93764224,"statfs":{"total":96636764160,"available":96014565376,"internally_reserved":0,"allocated":4833280,"data_stored":2295568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":126562,"internal_metadata":617222558},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706162,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1028892,"kb_used_data":4808,"kb_used_omap":110,"kb_used_meta":1023953,"kb_avail":93342948,"statfs":{"total":96636764160,"available":95583178752,"internally_reserved":0,"allocated":4923392,"data_stored":2913603,"data_compressed":3212,"data_compressed_allocated":40960,"data_compressed_original":581632,"omap_allocated":112971,"internal_metadata":1048528565},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706160,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1472508,"kb_used_data":4776,"kb_used_omap":124,"kb_used_meta":1467587,"kb_avail":92899332,"statfs":{"total":96636764160,"available":95128915968,"internally_reserved":0,"allocated":4890624,"data_stored":2885936,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":127209,"internal_metadata":1502809879},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T22:54:25.206 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-08-31T22:54:25.502 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:54:25.502 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-31T22:54:25.521 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1641,"stamp":"2024-08-31T22:54:25.395234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":96,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3109020,"kb_used_data":14304,"kb_used_omap":358,"kb_used_meta":3094297,"kb_avail":280006500,"statfs":{"total":289910292480,"available":286726656000,"internally_reserved":0,"allocated":14647296,"data_stored":8080720,"data_compressed":6354,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":366742,"internal_metadata":3168561002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-296236086,"num_objects":-123,"num_object_clones":-3,"num_object_copies":-246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-123,"num_whiteouts":-3,"num_read":-991920,"num_read_kb":-46245748,"num_write":-1199146,"num_write_kb":-23997177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-44315535,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002033"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":823,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.551816+0000","last_change":"2024-08-31T22:13:27.638168+0000","last_active":"2024-08-31T22:54:19.551816+0000","last_peered":"2024-08-31T22:54:19.551816+0000","last_clean":"2024-08-31T22:54:19.551816+0000","last_became_active":"2024-08-31T22:13:25.634194+0000","last_became_peered":"2024-08-31T22:13:25.634194+0000","last_unstale":"2024-08-31T22:54:19.551816+0000","last_undegraded":"2024-08-31T22:54:19.551816+0000","last_fullsized":"2024-08-31T22:54:19.551816+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T03:43:12.617984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024029400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":823,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.547368+0000","last_change":"2024-08-31T22:13:27.636754+0000","last_active":"2024-08-31T22:54:19.547368+0000","last_peered":"2024-08-31T22:54:19.547368+0000","last_clean":"2024-08-31T22:54:19.547368+0000","last_became_active":"2024-08-31T22:13:25.634366+0000","last_became_peered":"2024-08-31T22:13:25.634366+0000","last_unstale":"2024-08-31T22:54:19.547368+0000","last_undegraded":"2024-08-31T22:54:19.547368+0000","last_fullsized":"2024-08-31T22:54:19.547368+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:32:00.532608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001106084,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.559966+0000","last_change":"2024-08-31T22:13:27.637651+0000","last_active":"2024-08-31T22:54:19.559966+0000","last_peered":"2024-08-31T22:54:19.559966+0000","last_clean":"2024-08-31T22:54:19.559966+0000","last_became_active":"2024-08-31T22:13:25.634100+0000","last_became_peered":"2024-08-31T22:13:25.634100+0000","last_unstale":"2024-08-31T22:54:19.559966+0000","last_undegraded":"2024-08-31T22:54:19.559966+0000","last_fullsized":"2024-08-31T22:54:19.559966+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T09:42:20.559113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001870128,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.514353+0000","last_change":"2024-08-31T22:13:27.637511+0000","last_active":"2024-08-31T22:54:19.514353+0000","last_peered":"2024-08-31T22:54:19.514353+0000","last_clean":"2024-08-31T22:54:19.514353+0000","last_became_active":"2024-08-31T22:13:25.628362+0000","last_became_peered":"2024-08-31T22:13:25.628362+0000","last_unstale":"2024-08-31T22:54:19.514353+0000","last_undegraded":"2024-08-31T22:54:19.514353+0000","last_fullsized":"2024-08-31T22:54:19.514353+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T07:08:57.946059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001773806,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":826,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:21.990110+0000","last_change":"2024-08-31T22:13:27.640074+0000","last_active":"2024-08-31T22:54:21.990110+0000","last_peered":"2024-08-31T22:54:21.990110+0000","last_clean":"2024-08-31T22:54:21.990110+0000","last_became_active":"2024-08-31T22:13:25.628711+0000","last_became_peered":"2024-08-31T22:13:25.628711+0000","last_unstale":"2024-08-31T22:54:21.990110+0000","last_undegraded":"2024-08-31T22:54:21.990110+0000","last_fullsized":"2024-08-31T22:54:21.990110+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T08:42:16.373330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002200956,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":820,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-08-31T22:54:17.498849+0000","last_change":"2024-08-31T22:13:27.762539+0000","last_active":"2024-08-31T22:54:17.498849+0000","last_peered":"2024-08-31T22:54:17.498849+0000","last_clean":"2024-08-31T22:54:17.498849+0000","last_became_active":"2024-08-31T22:13:25.627201+0000","last_became_peered":"2024-08-31T22:13:25.627201+0000","last_unstale":"2024-08-31T22:54:17.498849+0000","last_undegraded":"2024-08-31T22:54:17.498849+0000","last_fullsized":"2024-08-31T22:54:17.498849+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T09:46:26.013807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054468599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":821,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-08-31T22:54:17.504974+0000","last_change":"2024-08-31T22:13:27.762844+0000","last_active":"2024-08-31T22:54:17.504974+0000","last_peered":"2024-08-31T22:54:17.504974+0000","last_clean":"2024-08-31T22:54:17.504974+0000","last_became_active":"2024-08-31T22:13:25.631027+0000","last_became_peered":"2024-08-31T22:13:25.631027+0000","last_unstale":"2024-08-31T22:54:17.504974+0000","last_undegraded":"2024-08-31T22:54:17.504974+0000","last_fullsized":"2024-08-31T22:54:17.504974+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:16:59.947661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060313200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":825,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.541603+0000","last_change":"2024-08-31T22:13:27.636882+0000","last_active":"2024-08-31T22:54:19.541603+0000","last_peered":"2024-08-31T22:54:19.541603+0000","last_clean":"2024-08-31T22:54:19.541603+0000","last_became_active":"2024-08-31T22:13:25.628867+0000","last_became_peered":"2024-08-31T22:13:25.628867+0000","last_unstale":"2024-08-31T22:54:19.541603+0000","last_undegraded":"2024-08-31T22:54:19.541603+0000","last_fullsized":"2024-08-31T22:54:19.541603+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:00:38.825010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001191278,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":953,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.515864+0000","last_change":"2024-08-31T22:13:21.644125+0000","last_active":"2024-08-31T22:54:19.515864+0000","last_peered":"2024-08-31T22:54:19.515864+0000","last_clean":"2024-08-31T22:54:19.515864+0000","last_became_active":"2024-08-31T22:13:21.641333+0000","last_became_peered":"2024-08-31T22:13:21.641333+0000","last_unstale":"2024-08-31T22:54:19.515864+0000","last_undegraded":"2024-08-31T22:54:19.515864+0000","last_fullsized":"2024-08-31T22:54:19.515864+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:20.589879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:20.589879+0000","last_clean_scrub_stamp":"2024-08-31T22:13:20.589879+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:38:29.946497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6354,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706164,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607616,"kb_used_data":4720,"kb_used_omap":123,"kb_used_meta":602756,"kb_avail":93764224,"statfs":{"total":96636764160,"available":96014565376,"internally_reserved":0,"allocated":4833280,"data_stored":2295568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":126562,"internal_metadata":617222558},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706163,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1028880,"kb_used_data":4792,"kb_used_omap":110,"kb_used_meta":1023953,"kb_avail":93342960,"statfs":{"total":96636764160,"available":95583191040,"internally_reserved":0,"allocated":4907008,"data_stored":2892576,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":112971,"internal_metadata":1048528565},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706161,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1472524,"kb_used_data":4792,"kb_used_omap":124,"kb_used_meta":1467587,"kb_avail":92899316,"statfs":{"total":96636764160,"available":95128899584,"internally_reserved":0,"allocated":4907008,"data_stored":2892576,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":127209,"internal_metadata":1502809879},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T22:54:25.522 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-08-31T22:54:25.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-08-31T22:54:25.826 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:54:25.826 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-31T22:54:25.845 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1641,"stamp":"2024-08-31T22:54:25.395234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":96,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3109020,"kb_used_data":14304,"kb_used_omap":358,"kb_used_meta":3094297,"kb_avail":280006500,"statfs":{"total":289910292480,"available":286726656000,"internally_reserved":0,"allocated":14647296,"data_stored":8080720,"data_compressed":6354,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":366742,"internal_metadata":3168561002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-296236086,"num_objects":-123,"num_object_clones":-3,"num_object_copies":-246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-123,"num_whiteouts":-3,"num_read":-991920,"num_read_kb":-46245748,"num_write":-1199146,"num_write_kb":-23997177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-44315535,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002033"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":823,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.551816+0000","last_change":"2024-08-31T22:13:27.638168+0000","last_active":"2024-08-31T22:54:19.551816+0000","last_peered":"2024-08-31T22:54:19.551816+0000","last_clean":"2024-08-31T22:54:19.551816+0000","last_became_active":"2024-08-31T22:13:25.634194+0000","last_became_peered":"2024-08-31T22:13:25.634194+0000","last_unstale":"2024-08-31T22:54:19.551816+0000","last_undegraded":"2024-08-31T22:54:19.551816+0000","last_fullsized":"2024-08-31T22:54:19.551816+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T03:43:12.617984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024029400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":823,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.547368+0000","last_change":"2024-08-31T22:13:27.636754+0000","last_active":"2024-08-31T22:54:19.547368+0000","last_peered":"2024-08-31T22:54:19.547368+0000","last_clean":"2024-08-31T22:54:19.547368+0000","last_became_active":"2024-08-31T22:13:25.634366+0000","last_became_peered":"2024-08-31T22:13:25.634366+0000","last_unstale":"2024-08-31T22:54:19.547368+0000","last_undegraded":"2024-08-31T22:54:19.547368+0000","last_fullsized":"2024-08-31T22:54:19.547368+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:32:00.532608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001106084,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.559966+0000","last_change":"2024-08-31T22:13:27.637651+0000","last_active":"2024-08-31T22:54:19.559966+0000","last_peered":"2024-08-31T22:54:19.559966+0000","last_clean":"2024-08-31T22:54:19.559966+0000","last_became_active":"2024-08-31T22:13:25.634100+0000","last_became_peered":"2024-08-31T22:13:25.634100+0000","last_unstale":"2024-08-31T22:54:19.559966+0000","last_undegraded":"2024-08-31T22:54:19.559966+0000","last_fullsized":"2024-08-31T22:54:19.559966+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T09:42:20.559113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001870128,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.514353+0000","last_change":"2024-08-31T22:13:27.637511+0000","last_active":"2024-08-31T22:54:19.514353+0000","last_peered":"2024-08-31T22:54:19.514353+0000","last_clean":"2024-08-31T22:54:19.514353+0000","last_became_active":"2024-08-31T22:13:25.628362+0000","last_became_peered":"2024-08-31T22:13:25.628362+0000","last_unstale":"2024-08-31T22:54:19.514353+0000","last_undegraded":"2024-08-31T22:54:19.514353+0000","last_fullsized":"2024-08-31T22:54:19.514353+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T07:08:57.946059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001773806,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":826,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:21.990110+0000","last_change":"2024-08-31T22:13:27.640074+0000","last_active":"2024-08-31T22:54:21.990110+0000","last_peered":"2024-08-31T22:54:21.990110+0000","last_clean":"2024-08-31T22:54:21.990110+0000","last_became_active":"2024-08-31T22:13:25.628711+0000","last_became_peered":"2024-08-31T22:13:25.628711+0000","last_unstale":"2024-08-31T22:54:21.990110+0000","last_undegraded":"2024-08-31T22:54:21.990110+0000","last_fullsized":"2024-08-31T22:54:21.990110+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T08:42:16.373330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002200956,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":820,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-08-31T22:54:17.498849+0000","last_change":"2024-08-31T22:13:27.762539+0000","last_active":"2024-08-31T22:54:17.498849+0000","last_peered":"2024-08-31T22:54:17.498849+0000","last_clean":"2024-08-31T22:54:17.498849+0000","last_became_active":"2024-08-31T22:13:25.627201+0000","last_became_peered":"2024-08-31T22:13:25.627201+0000","last_unstale":"2024-08-31T22:54:17.498849+0000","last_undegraded":"2024-08-31T22:54:17.498849+0000","last_fullsized":"2024-08-31T22:54:17.498849+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T09:46:26.013807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054468599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":821,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-08-31T22:54:17.504974+0000","last_change":"2024-08-31T22:13:27.762844+0000","last_active":"2024-08-31T22:54:17.504974+0000","last_peered":"2024-08-31T22:54:17.504974+0000","last_clean":"2024-08-31T22:54:17.504974+0000","last_became_active":"2024-08-31T22:13:25.631027+0000","last_became_peered":"2024-08-31T22:13:25.631027+0000","last_unstale":"2024-08-31T22:54:17.504974+0000","last_undegraded":"2024-08-31T22:54:17.504974+0000","last_fullsized":"2024-08-31T22:54:17.504974+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:16:59.947661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060313200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":825,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.541603+0000","last_change":"2024-08-31T22:13:27.636882+0000","last_active":"2024-08-31T22:54:19.541603+0000","last_peered":"2024-08-31T22:54:19.541603+0000","last_clean":"2024-08-31T22:54:19.541603+0000","last_became_active":"2024-08-31T22:13:25.628867+0000","last_became_peered":"2024-08-31T22:13:25.628867+0000","last_unstale":"2024-08-31T22:54:19.541603+0000","last_undegraded":"2024-08-31T22:54:19.541603+0000","last_fullsized":"2024-08-31T22:54:19.541603+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:00:38.825010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001191278,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":953,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.515864+0000","last_change":"2024-08-31T22:13:21.644125+0000","last_active":"2024-08-31T22:54:19.515864+0000","last_peered":"2024-08-31T22:54:19.515864+0000","last_clean":"2024-08-31T22:54:19.515864+0000","last_became_active":"2024-08-31T22:13:21.641333+0000","last_became_peered":"2024-08-31T22:13:21.641333+0000","last_unstale":"2024-08-31T22:54:19.515864+0000","last_undegraded":"2024-08-31T22:54:19.515864+0000","last_fullsized":"2024-08-31T22:54:19.515864+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:20.589879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:20.589879+0000","last_clean_scrub_stamp":"2024-08-31T22:13:20.589879+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:38:29.946497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6354,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706164,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607616,"kb_used_data":4720,"kb_used_omap":123,"kb_used_meta":602756,"kb_avail":93764224,"statfs":{"total":96636764160,"available":96014565376,"internally_reserved":0,"allocated":4833280,"data_stored":2295568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":126562,"internal_metadata":617222558},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706163,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1028880,"kb_used_data":4792,"kb_used_omap":110,"kb_used_meta":1023953,"kb_avail":93342960,"statfs":{"total":96636764160,"available":95583191040,"internally_reserved":0,"allocated":4907008,"data_stored":2892576,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":112971,"internal_metadata":1048528565},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706161,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1472524,"kb_used_data":4792,"kb_used_omap":124,"kb_used_meta":1467587,"kb_avail":92899316,"statfs":{"total":96636764160,"available":95128899584,"internally_reserved":0,"allocated":4907008,"data_stored":2892576,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":127209,"internal_metadata":1502809879},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T22:54:25.846 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-08-31T22:54:26.154 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:54:26.154 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":410,"fsid":"2454aa5a-0087-4f83-8364-979b8131b8af","created":"2024-08-31T22:13:08.964081+0000","modified":"2024-08-31T22:54:19.483730+0000","last_up_change":"2024-08-31T22:13:18.577125+0000","last_in_change":"2024-08-31T22:13:09.915598+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"octopus","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-31T22:13:20.588703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-31T22:13:24.457840+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2650a43d-a4d7-43a5-81c8-f8512897bad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3783604114},{"type":"v1","addr":"172.21.15.32:6817","nonce":3783604114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3783604114},{"type":"v1","addr":"172.21.15.32:6819","nonce":3783604114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3783604114},{"type":"v1","addr":"172.21.15.32:6823","nonce":3783604114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3783604114},{"type":"v1","addr":"172.21.15.32:6821","nonce":3783604114}]},"public_addr":"172.21.15.32:6817/3783604114","cluster_addr":"172.21.15.32:6819/3783604114","heartbeat_back_addr":"172.21.15.32:6823/3783604114","heartbeat_front_addr":"172.21.15.32:6821/3783604114","state":["exists","up"]},{"osd":1,"uuid":"330b62c6-fe47-4b1e-b61c-7abf9c9fa380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":201348119},{"type":"v1","addr":"172.21.15.32:6801","nonce":201348119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":201348119},{"type":"v1","addr":"172.21.15.32:6803","nonce":201348119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":201348119},{"type":"v1","addr":"172.21.15.32:6807","nonce":201348119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":201348119},{"type":"v1","addr":"172.21.15.32:6805","nonce":201348119}]},"public_addr":"172.21.15.32:6801/201348119","cluster_addr":"172.21.15.32:6803/201348119","heartbeat_back_addr":"172.21.15.32:6807/201348119","heartbeat_front_addr":"172.21.15.32:6805/201348119","state":["exists","up"]},{"osd":2,"uuid":"47505ad1-7bd0-4a0f-ab31-1cd6778fe9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":4129613497},{"type":"v1","addr":"172.21.15.32:6809","nonce":4129613497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":4129613497},{"type":"v1","addr":"172.21.15.32:6811","nonce":4129613497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":4129613497},{"type":"v1","addr":"172.21.15.32:6815","nonce":4129613497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":4129613497},{"type":"v1","addr":"172.21.15.32:6813","nonce":4129613497}]},"public_addr":"172.21.15.32:6809/4129613497","cluster_addr":"172.21.15.32:6811/4129613497","heartbeat_back_addr":"172.21.15.32:6815/4129613497","heartbeat_front_addr":"172.21.15.32:6813/4129613497","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:13:16.673817+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:13:15.983995+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:13:16.448905+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/420110542":"2024-08-31T23:54:13.896970+0000","172.21.15.32:0/3092508363":"2024-08-31T23:53:21.176975+0000","172.21.15.32:0/3041263916":"2024-08-31T23:51:52.598647+0000","172.21.15.32:0/1937917870":"2024-08-31T23:50:33.092062+0000","172.21.15.32:0/539442562":"2024-08-31T23:49:53.938908+0000","172.21.15.32:0/1102577476":"2024-08-31T23:49:12.335766+0000","172.21.15.32:0/2457483155":"2024-08-31T23:48:31.203288+0000","172.21.15.32:0/3606187552":"2024-08-31T23:47:52.597524+0000","172.21.15.32:0/1282338676":"2024-08-31T23:41:02.187500+0000","172.21.15.32:0/565561015":"2024-08-31T23:30:42.993394+0000","172.21.15.32:0/3568535500":"2024-08-31T23:40:42.170122+0000","172.21.15.32:0/2973996612":"2024-08-31T23:41:50.830823+0000","172.21.15.32:0/1407953486":"2024-08-31T23:36:00.684280+0000","172.21.15.32:0/3127072232":"2024-08-31T23:15:40.563468+0000","172.21.15.32:0/3405024924":"2024-08-31T23:45:59.152849+0000","172.21.15.32:0/3975514315":"2024-08-31T23:35:13.854046+0000","172.21.15.32:0/2565425486":"2024-08-31T23:43:09.817605+0000","172.21.15.32:0/2683629818":"2024-08-31T23:34:35.678069+0000","172.21.15.32:0/1552195651":"2024-08-31T23:41:44.875664+0000","172.21.15.32:0/1924880922":"2024-08-31T23:44:56.316998+0000","172.21.15.32:0/3772272028":"2024-08-31T23:37:49.355209+0000","172.21.15.32:0/1228652023":"2024-08-31T23:52:34.387918+0000","172.21.15.32:0/2118842236":"2024-08-31T23:17:27.461500+0000","172.21.15.32:0/2160439951":"2024-08-31T23:42:48.923359+0000","172.21.15.32:0/2509684986":"2024-08-31T23:33:33.419887+0000","172.21.15.32:0/746505778":"2024-08-31T23:19:06.653392+0000","172.21.15.32:0/4030316002":"2024-08-31T23:45:08.755864+0000","172.21.15.32:0/2053746109":"2024-08-31T23:38:37.089659+0000","172.21.15.32:0/1652283377":"2024-08-31T23:44:29.226315+0000","172.21.15.32:0/4255745503":"2024-08-31T23:36:40.621566+0000","172.21.15.32:0/1448280902":"2024-08-31T23:28:07.223087+0000","172.21.15.32:0/1299217857":"2024-08-31T23:43:17.375594+0000","172.21.15.32:0/3417262013":"2024-08-31T23:25:11.837218+0000","172.21.15.32:0/2941800":"2024-08-31T23:36:55.680295+0000","172.21.15.32:0/1318022240":"2024-08-31T23:29:34.173585+0000","172.21.15.32:0/656247971":"2024-08-31T23:42:11.153315+0000","172.21.15.32:0/2022438414":"2024-08-31T23:42:16.173059+0000","172.21.15.32:0/1612543675":"2024-08-31T23:42:37.100040+0000","172.21.15.32:0/1404804511":"2024-08-31T23:26:09.405426+0000","172.21.15.32:0/1709697431":"2024-08-31T23:42:42.974579+0000","172.21.15.32:0/548240184":"2024-08-31T23:33:43.237176+0000","172.21.15.32:0/286863706":"2024-08-31T23:43:27.899887+0000","172.21.15.32:0/1137889185":"2024-08-31T23:42:21.267761+0000","172.21.15.32:0/2964084182":"2024-08-31T23:44:07.083952+0000","172.21.15.32:0/2743330664":"2024-08-31T23:45:03.103474+0000","172.21.15.32:0/3991116214":"2024-08-31T23:33:50.192690+0000","172.21.15.32:0/1226917565":"2024-08-31T23:45:42.918274+0000","172.21.15.32:0/1558474990":"2024-08-31T23:45:14.226735+0000","172.21.15.32:0/1576170748":"2024-08-31T23:43:39.664277+0000","172.21.15.32:0/689671948":"2024-08-31T23:35:28.886947+0000","172.21.15.32:0/2817102365":"2024-08-31T23:43:50.929184+0000","172.21.15.32:0/1577602192":"2024-08-31T23:51:12.082056+0000","172.21.15.32:0/1479864240":"2024-08-31T23:33:10.121151+0000","172.21.15.32:0/10651016":"2024-08-31T23:44:00.743440+0000","172.21.15.32:0/3113850320":"2024-08-31T23:34:30.498701+0000","172.21.15.32:0/379495491":"2024-08-31T23:44:13.058312+0000","172.21.15.32:0/1944526858":"2024-08-31T23:21:04.958706+0000","172.21.15.32:0/1718068696":"2024-08-31T23:43:33.670113+0000","172.21.15.32:0/1861120950":"2024-08-31T23:44:19.228660+0000","172.21.15.32:0/1597569381":"2024-08-31T23:45:38.227076+0000","172.21.15.32:0/608791703":"2024-08-31T23:44:34.894499+0000","172.21.15.32:0/842306197":"2024-08-31T23:23:03.895757+0000","172.21.15.32:0/3530814651":"2024-08-31T23:42:58.798634+0000","172.21.15.32:0/323556361":"2024-08-31T23:44:40.221794+0000","172.21.15.32:0/1753077816":"2024-08-31T23:40:36.609663+0000","172.21.15.32:0/667028845":"2024-08-31T23:44:50.012511+0000","172.21.15.32:0/2164135235":"2024-08-31T23:45:18.860107+0000","172.21.15.32:0/753374140":"2024-08-31T23:41:33.168283+0000","172.21.15.32:0/1633632482":"2024-08-31T23:45:25.215421+0000","172.21.15.32:0/1673601874":"2024-08-31T23:45:31.416036+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T22:54:27.175 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-08-31T22:54:27.175 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-08-31T22:54:27.340 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-31T22:54:27.340 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-31T22:54:27.340 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-31T22:54:27.357 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-08-31T22:54:27.657 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 0 to deep-scrub 2024-08-31T22:54:27.679 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-08-31T22:54:27.679 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-08-31T22:54:27.831 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-31T22:54:27.831 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-31T22:54:27.831 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-31T22:54:27.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-08-31T22:54:28.147 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 1 to deep-scrub 2024-08-31T22:54:28.165 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-08-31T22:54:28.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-08-31T22:54:28.317 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-31T22:54:28.318 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-31T22:54:28.318 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-31T22:54:28.333 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-08-31T22:54:28.631 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 2 to deep-scrub 2024-08-31T22:54:28.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-08-31T22:54:28.937 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:54:28.937 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-31T22:54:28.955 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1642,"stamp":"2024-08-31T22:54:27.395725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3109036,"kb_used_data":14316,"kb_used_omap":358,"kb_used_meta":3094297,"kb_avail":280006484,"statfs":{"total":289910292480,"available":286726639616,"internally_reserved":0,"allocated":14659584,"data_stored":8087341,"data_compressed":6354,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":366742,"internal_metadata":3168561002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-6,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":0,"num_read":-2536,"num_read_kb":-2525,"num_write":-168,"num_write_kb":-165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.908573"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":823,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.551816+0000","last_change":"2024-08-31T22:13:27.638168+0000","last_active":"2024-08-31T22:54:19.551816+0000","last_peered":"2024-08-31T22:54:19.551816+0000","last_clean":"2024-08-31T22:54:19.551816+0000","last_became_active":"2024-08-31T22:13:25.634194+0000","last_became_peered":"2024-08-31T22:13:25.634194+0000","last_unstale":"2024-08-31T22:54:19.551816+0000","last_undegraded":"2024-08-31T22:54:19.551816+0000","last_fullsized":"2024-08-31T22:54:19.551816+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T03:43:12.617984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024029400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":823,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.547368+0000","last_change":"2024-08-31T22:13:27.636754+0000","last_active":"2024-08-31T22:54:19.547368+0000","last_peered":"2024-08-31T22:54:19.547368+0000","last_clean":"2024-08-31T22:54:19.547368+0000","last_became_active":"2024-08-31T22:13:25.634366+0000","last_became_peered":"2024-08-31T22:13:25.634366+0000","last_unstale":"2024-08-31T22:54:19.547368+0000","last_undegraded":"2024-08-31T22:54:19.547368+0000","last_fullsized":"2024-08-31T22:54:19.547368+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:32:00.532608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001106084,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.559966+0000","last_change":"2024-08-31T22:13:27.637651+0000","last_active":"2024-08-31T22:54:19.559966+0000","last_peered":"2024-08-31T22:54:19.559966+0000","last_clean":"2024-08-31T22:54:19.559966+0000","last_became_active":"2024-08-31T22:13:25.634100+0000","last_became_peered":"2024-08-31T22:13:25.634100+0000","last_unstale":"2024-08-31T22:54:19.559966+0000","last_undegraded":"2024-08-31T22:54:19.559966+0000","last_fullsized":"2024-08-31T22:54:19.559966+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T09:42:20.559113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001870128,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.514353+0000","last_change":"2024-08-31T22:13:27.637511+0000","last_active":"2024-08-31T22:54:19.514353+0000","last_peered":"2024-08-31T22:54:19.514353+0000","last_clean":"2024-08-31T22:54:19.514353+0000","last_became_active":"2024-08-31T22:13:25.628362+0000","last_became_peered":"2024-08-31T22:13:25.628362+0000","last_unstale":"2024-08-31T22:54:19.514353+0000","last_undegraded":"2024-08-31T22:54:19.514353+0000","last_fullsized":"2024-08-31T22:54:19.514353+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T07:08:57.946059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001773806,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":826,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:21.990110+0000","last_change":"2024-08-31T22:13:27.640074+0000","last_active":"2024-08-31T22:54:21.990110+0000","last_peered":"2024-08-31T22:54:21.990110+0000","last_clean":"2024-08-31T22:54:21.990110+0000","last_became_active":"2024-08-31T22:13:25.628711+0000","last_became_peered":"2024-08-31T22:13:25.628711+0000","last_unstale":"2024-08-31T22:54:21.990110+0000","last_undegraded":"2024-08-31T22:54:21.990110+0000","last_fullsized":"2024-08-31T22:54:21.990110+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T08:42:16.373330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002200956,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":822,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:21.125706+0000","last_change":"2024-08-31T22:13:27.762539+0000","last_active":"2024-08-31T22:54:21.125706+0000","last_peered":"2024-08-31T22:54:21.125706+0000","last_clean":"2024-08-31T22:54:21.125706+0000","last_became_active":"2024-08-31T22:13:25.627201+0000","last_became_peered":"2024-08-31T22:13:25.627201+0000","last_unstale":"2024-08-31T22:54:21.125706+0000","last_undegraded":"2024-08-31T22:54:21.125706+0000","last_fullsized":"2024-08-31T22:54:21.125706+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T09:46:26.013807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054468599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":823,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:21.128172+0000","last_change":"2024-08-31T22:13:27.762844+0000","last_active":"2024-08-31T22:54:21.128172+0000","last_peered":"2024-08-31T22:54:21.128172+0000","last_clean":"2024-08-31T22:54:21.128172+0000","last_became_active":"2024-08-31T22:13:25.631027+0000","last_became_peered":"2024-08-31T22:13:25.631027+0000","last_unstale":"2024-08-31T22:54:21.128172+0000","last_undegraded":"2024-08-31T22:54:21.128172+0000","last_fullsized":"2024-08-31T22:54:21.128172+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:16:59.947661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060313200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":825,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.541603+0000","last_change":"2024-08-31T22:13:27.636882+0000","last_active":"2024-08-31T22:54:19.541603+0000","last_peered":"2024-08-31T22:54:19.541603+0000","last_clean":"2024-08-31T22:54:19.541603+0000","last_became_active":"2024-08-31T22:13:25.628867+0000","last_became_peered":"2024-08-31T22:13:25.628867+0000","last_unstale":"2024-08-31T22:54:19.541603+0000","last_undegraded":"2024-08-31T22:54:19.541603+0000","last_fullsized":"2024-08-31T22:54:19.541603+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:24.610842+0000","last_clean_scrub_stamp":"2024-08-31T22:13:24.610842+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:00:38.825010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001191278,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":953,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.515864+0000","last_change":"2024-08-31T22:13:21.644125+0000","last_active":"2024-08-31T22:54:19.515864+0000","last_peered":"2024-08-31T22:54:19.515864+0000","last_clean":"2024-08-31T22:54:19.515864+0000","last_became_active":"2024-08-31T22:13:21.641333+0000","last_became_peered":"2024-08-31T22:13:21.641333+0000","last_unstale":"2024-08-31T22:54:19.515864+0000","last_undegraded":"2024-08-31T22:54:19.515864+0000","last_fullsized":"2024-08-31T22:54:19.515864+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:13:20.589879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:13:20.589879+0000","last_clean_scrub_stamp":"2024-08-31T22:13:20.589879+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:38:29.946497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6354,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706165,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607632,"kb_used_data":4732,"kb_used_omap":123,"kb_used_meta":602756,"kb_avail":93764208,"statfs":{"total":96636764160,"available":96014548992,"internally_reserved":0,"allocated":4845568,"data_stored":2302189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":126562,"internal_metadata":617222558},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706163,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1028880,"kb_used_data":4792,"kb_used_omap":110,"kb_used_meta":1023953,"kb_avail":93342960,"statfs":{"total":96636764160,"available":95583191040,"internally_reserved":0,"allocated":4907008,"data_stored":2892576,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":112971,"internal_metadata":1048528565},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706161,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1472524,"kb_used_data":4792,"kb_used_omap":124,"kb_used_meta":1467587,"kb_avail":92899316,"statfs":{"total":96636764160,"available":95128899584,"internally_reserved":0,"allocated":4907008,"data_stored":2892576,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":127209,"internal_metadata":1502809879},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T22:54:28.958 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-08-31T22:13:24.610842+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=13, tm_sec=24, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=54, tm_sec=26, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T22:54:28.958 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-08-31T22:13:24.610842+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=13, tm_sec=24, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=54, tm_sec=26, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T22:54:28.959 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-08-31T22:13:24.610842+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=13, tm_sec=24, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=54, tm_sec=26, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T22:54:28.959 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-08-31T22:13:24.610842+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=13, tm_sec=24, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=54, tm_sec=26, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T22:54:28.959 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-08-31T22:13:24.610842+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=13, tm_sec=24, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=54, tm_sec=26, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T22:54:28.959 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-08-31T22:13:24.610842+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=13, tm_sec=24, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=54, tm_sec=26, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T22:54:28.959 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-08-31T22:13:24.610842+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=13, tm_sec=24, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=54, tm_sec=26, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T22:54:28.959 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-08-31T22:13:24.610842+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=13, tm_sec=24, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=54, tm_sec=26, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T22:54:28.959 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-08-31T22:13:20.589879+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=13, tm_sec=20, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=54, tm_sec=26, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T22:54:28.959 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-08-31T22:54:48.960 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-08-31T22:54:49.275 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-31T22:54:49.275 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-31T22:54:49.294 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1652,"stamp":"2024-08-31T22:54:47.400504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3109020,"kb_used_data":14316,"kb_used_omap":358,"kb_used_meta":3094297,"kb_avail":280006500,"statfs":{"total":289910292480,"available":286726656000,"internally_reserved":0,"allocated":14659584,"data_stored":8087341,"data_compressed":6354,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":366742,"internal_metadata":3168561002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002876"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":833,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:31.471224+0000","last_change":"2024-08-31T22:54:31.471224+0000","last_active":"2024-08-31T22:54:31.471224+0000","last_peered":"2024-08-31T22:54:31.471224+0000","last_clean":"2024-08-31T22:54:31.471224+0000","last_became_active":"2024-08-31T22:13:25.634194+0000","last_became_peered":"2024-08-31T22:13:25.634194+0000","last_unstale":"2024-08-31T22:54:31.471224+0000","last_undegraded":"2024-08-31T22:54:31.471224+0000","last_fullsized":"2024-08-31T22:54:31.471224+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:54:31.470952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:54:31.470952+0000","last_clean_scrub_stamp":"2024-08-31T22:54:31.470952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:36:28.831027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024029400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":833,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:33.445462+0000","last_change":"2024-08-31T22:54:33.445462+0000","last_active":"2024-08-31T22:54:33.445462+0000","last_peered":"2024-08-31T22:54:33.445462+0000","last_clean":"2024-08-31T22:54:33.445462+0000","last_became_active":"2024-08-31T22:13:25.634366+0000","last_became_peered":"2024-08-31T22:13:25.634366+0000","last_unstale":"2024-08-31T22:54:33.445462+0000","last_undegraded":"2024-08-31T22:54:33.445462+0000","last_fullsized":"2024-08-31T22:54:33.445462+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:54:33.445216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:54:33.445216+0000","last_clean_scrub_stamp":"2024-08-31T22:54:33.445216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:12:32.357670+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001106084,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":833,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:32.429862+0000","last_change":"2024-08-31T22:54:32.429914+0000","last_active":"2024-08-31T22:54:32.429862+0000","last_peered":"2024-08-31T22:54:32.429862+0000","last_clean":"2024-08-31T22:54:32.429862+0000","last_became_active":"2024-08-31T22:13:25.634100+0000","last_became_peered":"2024-08-31T22:13:25.634100+0000","last_unstale":"2024-08-31T22:54:32.429862+0000","last_undegraded":"2024-08-31T22:54:32.429862+0000","last_fullsized":"2024-08-31T22:54:32.429862+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:54:32.429672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:54:32.429672+0000","last_clean_scrub_stamp":"2024-08-31T22:54:32.429672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T09:54:31.991324+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001870128,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":833,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:30.433873+0000","last_change":"2024-08-31T22:54:30.433974+0000","last_active":"2024-08-31T22:54:30.433873+0000","last_peered":"2024-08-31T22:54:30.433873+0000","last_clean":"2024-08-31T22:54:30.433873+0000","last_became_active":"2024-08-31T22:13:25.628362+0000","last_became_peered":"2024-08-31T22:13:25.628362+0000","last_unstale":"2024-08-31T22:54:30.433873+0000","last_undegraded":"2024-08-31T22:54:30.433873+0000","last_fullsized":"2024-08-31T22:54:30.433873+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:54:30.433561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:54:30.433561+0000","last_clean_scrub_stamp":"2024-08-31T22:54:30.433561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T03:02:10.142386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001773806,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"410'3","reported_seq":836,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:28.468490+0000","last_change":"2024-08-31T22:54:28.468536+0000","last_active":"2024-08-31T22:54:28.468490+0000","last_peered":"2024-08-31T22:54:28.468490+0000","last_clean":"2024-08-31T22:54:28.468490+0000","last_became_active":"2024-08-31T22:13:25.628711+0000","last_became_peered":"2024-08-31T22:13:25.628711+0000","last_unstale":"2024-08-31T22:54:28.468490+0000","last_undegraded":"2024-08-31T22:54:28.468490+0000","last_fullsized":"2024-08-31T22:54:28.468490+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'3","last_scrub_stamp":"2024-08-31T22:54:28.468379+0000","last_deep_scrub":"410'3","last_deep_scrub_stamp":"2024-08-31T22:54:28.468379+0000","last_clean_scrub_stamp":"2024-08-31T22:54:28.468379+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T01:15:25.186847+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002200956,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":831,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:29.449407+0000","last_change":"2024-08-31T22:54:29.449458+0000","last_active":"2024-08-31T22:54:29.449407+0000","last_peered":"2024-08-31T22:54:29.449407+0000","last_clean":"2024-08-31T22:54:29.449407+0000","last_became_active":"2024-08-31T22:13:25.627201+0000","last_became_peered":"2024-08-31T22:13:25.627201+0000","last_unstale":"2024-08-31T22:54:29.449407+0000","last_undegraded":"2024-08-31T22:54:29.449407+0000","last_fullsized":"2024-08-31T22:54:29.449407+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:54:29.449217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:54:29.449217+0000","last_clean_scrub_stamp":"2024-08-31T22:54:29.449217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:16:29.363712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054468599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":832,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:30.436037+0000","last_change":"2024-08-31T22:54:30.436093+0000","last_active":"2024-08-31T22:54:30.436037+0000","last_peered":"2024-08-31T22:54:30.436037+0000","last_clean":"2024-08-31T22:54:30.436037+0000","last_became_active":"2024-08-31T22:13:25.631027+0000","last_became_peered":"2024-08-31T22:13:25.631027+0000","last_unstale":"2024-08-31T22:54:30.436037+0000","last_undegraded":"2024-08-31T22:54:30.436037+0000","last_fullsized":"2024-08-31T22:54:30.436037+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:54:30.435842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:54:30.435842+0000","last_clean_scrub_stamp":"2024-08-31T22:54:30.435842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T07:01:44.032591+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00060313200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"410'2","reported_seq":835,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:29.420423+0000","last_change":"2024-08-31T22:54:29.420490+0000","last_active":"2024-08-31T22:54:29.420423+0000","last_peered":"2024-08-31T22:54:29.420423+0000","last_clean":"2024-08-31T22:54:29.420423+0000","last_became_active":"2024-08-31T22:13:25.628867+0000","last_became_peered":"2024-08-31T22:13:25.628867+0000","last_unstale":"2024-08-31T22:54:29.420423+0000","last_undegraded":"2024-08-31T22:54:29.420423+0000","last_fullsized":"2024-08-31T22:54:29.420423+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'2","last_scrub_stamp":"2024-08-31T22:54:29.420225+0000","last_deep_scrub":"410'2","last_deep_scrub_stamp":"2024-08-31T22:54:29.420225+0000","last_clean_scrub_stamp":"2024-08-31T22:54:29.420225+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T05:01:20.224082+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001191278,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"410'99","reported_seq":964,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-08-31T22:54:28.463686+0000","last_change":"2024-08-31T22:54:28.463720+0000","last_active":"2024-08-31T22:54:28.463686+0000","last_peered":"2024-08-31T22:54:28.463686+0000","last_clean":"2024-08-31T22:54:28.463686+0000","last_became_active":"2024-08-31T22:13:21.641333+0000","last_became_peered":"2024-08-31T22:13:21.641333+0000","last_unstale":"2024-08-31T22:54:28.463686+0000","last_undegraded":"2024-08-31T22:54:28.463686+0000","last_fullsized":"2024-08-31T22:54:28.463686+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'99","last_scrub_stamp":"2024-08-31T22:54:28.463584+0000","last_deep_scrub":"410'99","last_deep_scrub_stamp":"2024-08-31T22:54:28.463584+0000","last_clean_scrub_stamp":"2024-08-31T22:54:28.463584+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T01:19:39.335913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6354,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706169,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607628,"kb_used_data":4732,"kb_used_omap":123,"kb_used_meta":602756,"kb_avail":93764212,"statfs":{"total":96636764160,"available":96014553088,"internally_reserved":0,"allocated":4845568,"data_stored":2302189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":126562,"internal_metadata":617222558},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706167,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1028872,"kb_used_data":4792,"kb_used_omap":110,"kb_used_meta":1023953,"kb_avail":93342968,"statfs":{"total":96636764160,"available":95583199232,"internally_reserved":0,"allocated":4907008,"data_stored":2892576,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":112971,"internal_metadata":1048528565},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706165,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1472520,"kb_used_data":4792,"kb_used_omap":124,"kb_used_meta":1467587,"kb_avail":92899320,"statfs":{"total":96636764160,"available":95128903680,"internally_reserved":0,"allocated":4907008,"data_stored":2892576,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":127209,"internal_metadata":1502809879},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3177,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T22:54:49.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-08-31T22:54:49.634 INFO:teuthology.misc:Shutting down mds daemons... 2024-08-31T22:54:49.635 INFO:teuthology.misc:Shutting down osd daemons... 2024-08-31T22:54:49.635 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-08-31T22:54:49.635 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T22:54:50.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2024-08-31T22:54:55.737 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-08-31T22:54:55.737 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-08-31T22:54:55.737 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T22:54:56.087 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2024-08-31T22:55:01.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~0s 2024-08-31T22:55:01.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 has been restored 2024-08-31T22:55:01.839 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-08-31T22:55:01.840 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-08-31T22:55:01.840 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T22:55:06.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 is failed for ~0s 2024-08-31T22:55:06.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 has been restored 2024-08-31T22:55:07.942 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-08-31T22:55:07.942 INFO:teuthology.misc:Shutting down mgr daemons... 2024-08-31T22:55:07.942 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-08-31T22:55:07.942 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T22:55:07.981 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-08-31T22:55:07.982 INFO:teuthology.misc:Shutting down mon daemons... 2024-08-31T22:55:07.982 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-08-31T22:55:07.982 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T22:55:08.011 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-08-31T22:55:08.011 INFO:tasks.ceph:Checking cluster log for badness... 2024-08-31T22:55:08.012 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-08-31T22:55:08.155 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi032.front.sepia.ceph.com 2024-08-31T22:55:08.155 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-08-31T22:55:08.492 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi032.front.sepia.ceph.com 2024-08-31T22:55:08.492 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-08-31T22:55:08.781 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi032.front.sepia.ceph.com 2024-08-31T22:55:08.781 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-08-31T22:55:09.143 INFO:tasks.ceph:Archiving mon data... 2024-08-31T22:55:09.145 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/teuthology-2024-08-31_21:16:02-rbd-squid-distro-default-smithi/7883608/data/cluster2.mon.a.tgz 2024-08-31T22:55:09.145 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-08-31T22:55:09.151 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.1FhQpeFnPu 2024-08-31T22:55:09.151 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.1FhQpeFnPu 2024-08-31T22:55:09.320 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.1FhQpeFnPu 2024-08-31T22:55:09.401 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.1FhQpeFnPu is 503KB 2024-08-31T22:55:09.465 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.1FhQpeFnPu 2024-08-31T22:55:09.471 INFO:tasks.ceph:Cleaning ceph cluster... 2024-08-31T22:55:09.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-08-31T22:55:09.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-08-31T22:55:09.658 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-08-31T22:55:09.666 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-08-31T22:55:09.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-08-31T22:55:10.191 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:55:10.191 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-08-31T22:55:10.231 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":1614,"stamp":"2024-08-31T22:55:09.237805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4745268,"kb_used_data":8376,"kb_used_omap":645,"kb_used_meta":4736186,"kb_avail":278370252,"statfs":{"total":289910292480,"available":285051138048,"internally_reserved":0,"allocated":8577024,"data_stored":5625829,"data_compressed":6360,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":661497,"internal_metadata":4849854471},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002987"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":688,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.268404+0000","last_change":"2024-08-31T22:12:50.394482+0000","last_active":"2024-08-31T22:54:20.268404+0000","last_peered":"2024-08-31T22:54:20.268404+0000","last_clean":"2024-08-31T22:54:20.268404+0000","last_became_active":"2024-08-31T22:12:48.393530+0000","last_became_peered":"2024-08-31T22:12:48.393530+0000","last_unstale":"2024-08-31T22:54:20.268404+0000","last_undegraded":"2024-08-31T22:54:20.268404+0000","last_fullsized":"2024-08-31T22:54:20.268404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:53:59.887015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021475320000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":688,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.270604+0000","last_change":"2024-08-31T22:12:50.394918+0000","last_active":"2024-08-31T22:54:20.270604+0000","last_peered":"2024-08-31T22:54:20.270604+0000","last_clean":"2024-08-31T22:54:20.270604+0000","last_became_active":"2024-08-31T22:12:48.390967+0000","last_became_peered":"2024-08-31T22:12:48.390967+0000","last_unstale":"2024-08-31T22:54:20.270604+0000","last_undegraded":"2024-08-31T22:54:20.270604+0000","last_fullsized":"2024-08-31T22:54:20.270604+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T07:18:29.001799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002301787,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":689,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.270653+0000","last_change":"2024-08-31T22:12:50.393340+0000","last_active":"2024-08-31T22:54:20.270653+0000","last_peered":"2024-08-31T22:54:20.270653+0000","last_clean":"2024-08-31T22:54:20.270653+0000","last_became_active":"2024-08-31T22:12:48.393810+0000","last_became_peered":"2024-08-31T22:12:48.393810+0000","last_unstale":"2024-08-31T22:54:20.270653+0000","last_undegraded":"2024-08-31T22:54:20.270653+0000","last_fullsized":"2024-08-31T22:54:20.270653+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:37:46.050434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011422590000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":689,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.268077+0000","last_change":"2024-08-31T22:12:50.394198+0000","last_active":"2024-08-31T22:54:20.268077+0000","last_peered":"2024-08-31T22:54:20.268077+0000","last_clean":"2024-08-31T22:54:20.268077+0000","last_became_active":"2024-08-31T22:12:48.395901+0000","last_became_peered":"2024-08-31T22:12:48.395901+0000","last_unstale":"2024-08-31T22:54:20.268077+0000","last_undegraded":"2024-08-31T22:54:20.268077+0000","last_fullsized":"2024-08-31T22:54:20.268077+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T02:29:23.840873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018848599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":690,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:18.977795+0000","last_change":"2024-08-31T22:12:50.400538+0000","last_active":"2024-08-31T22:54:18.977795+0000","last_peered":"2024-08-31T22:54:18.977795+0000","last_clean":"2024-08-31T22:54:18.977795+0000","last_became_active":"2024-08-31T22:12:48.389795+0000","last_became_peered":"2024-08-31T22:12:48.389795+0000","last_unstale":"2024-08-31T22:54:18.977795+0000","last_undegraded":"2024-08-31T22:54:18.977795+0000","last_fullsized":"2024-08-31T22:54:18.977795+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T01:35:29.294687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023100410000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":689,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.670977+0000","last_change":"2024-08-31T22:12:51.173642+0000","last_active":"2024-08-31T22:54:19.670977+0000","last_peered":"2024-08-31T22:54:19.670977+0000","last_clean":"2024-08-31T22:54:19.670977+0000","last_became_active":"2024-08-31T22:12:48.391597+0000","last_became_peered":"2024-08-31T22:12:48.391597+0000","last_unstale":"2024-08-31T22:54:19.670977+0000","last_undegraded":"2024-08-31T22:54:19.670977+0000","last_fullsized":"2024-08-31T22:54:19.670977+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:22:09.506410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051578099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":690,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.669859+0000","last_change":"2024-08-31T22:12:51.173571+0000","last_active":"2024-08-31T22:54:19.669859+0000","last_peered":"2024-08-31T22:54:19.669859+0000","last_clean":"2024-08-31T22:54:19.669859+0000","last_became_active":"2024-08-31T22:12:48.393856+0000","last_became_peered":"2024-08-31T22:12:48.393856+0000","last_unstale":"2024-08-31T22:54:19.669859+0000","last_undegraded":"2024-08-31T22:54:19.669859+0000","last_fullsized":"2024-08-31T22:54:19.669859+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:15:04.841041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046582699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":690,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.273404+0000","last_change":"2024-08-31T22:12:50.393581+0000","last_active":"2024-08-31T22:54:20.273404+0000","last_peered":"2024-08-31T22:54:20.273404+0000","last_clean":"2024-08-31T22:54:20.273404+0000","last_became_active":"2024-08-31T22:12:48.393780+0000","last_became_peered":"2024-08-31T22:12:48.393780+0000","last_unstale":"2024-08-31T22:54:20.273404+0000","last_undegraded":"2024-08-31T22:54:20.273404+0000","last_fullsized":"2024-08-31T22:54:20.273404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T01:27:52.811385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013191050000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":818,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.266663+0000","last_change":"2024-08-31T22:12:44.358595+0000","last_active":"2024-08-31T22:54:20.266663+0000","last_peered":"2024-08-31T22:54:20.266663+0000","last_clean":"2024-08-31T22:54:20.266663+0000","last_became_active":"2024-08-31T22:12:44.357729+0000","last_became_peered":"2024-08-31T22:12:44.357729+0000","last_unstale":"2024-08-31T22:54:20.266663+0000","last_undegraded":"2024-08-31T22:54:20.266663+0000","last_fullsized":"2024-08-31T22:54:20.266663+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:43.340061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:43.340061+0000","last_clean_scrub_stamp":"2024-08-31T22:12:43.340061+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:14:47.042415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6360,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706176,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1754900,"kb_used_data":2752,"kb_used_omap":124,"kb_used_meta":1752003,"kb_avail":92616940,"statfs":{"total":96636764160,"available":94839746560,"internally_reserved":0,"allocated":2818048,"data_stored":1481685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":127899,"internal_metadata":1794051173},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706177,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1439440,"kb_used_data":2812,"kb_used_omap":256,"kb_used_meta":1436351,"kb_avail":92932400,"statfs":{"total":96636764160,"available":95162777600,"internally_reserved":0,"allocated":2879488,"data_stored":2072072,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":263067,"internal_metadata":1470823525},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738882,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1550928,"kb_used_data":2812,"kb_used_omap":264,"kb_used_meta":1547831,"kb_avail":92820912,"statfs":{"total":96636764160,"available":95048613888,"internally_reserved":0,"allocated":2879488,"data_stored":2072072,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":270531,"internal_metadata":1584979773},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T22:55:10.232 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-08-31T22:55:10.534 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:55:10.534 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-08-31T22:55:10.553 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":1614,"stamp":"2024-08-31T22:55:09.237805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4745268,"kb_used_data":8376,"kb_used_omap":645,"kb_used_meta":4736186,"kb_avail":278370252,"statfs":{"total":289910292480,"available":285051138048,"internally_reserved":0,"allocated":8577024,"data_stored":5625829,"data_compressed":6360,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":661497,"internal_metadata":4849854471},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002987"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":688,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.268404+0000","last_change":"2024-08-31T22:12:50.394482+0000","last_active":"2024-08-31T22:54:20.268404+0000","last_peered":"2024-08-31T22:54:20.268404+0000","last_clean":"2024-08-31T22:54:20.268404+0000","last_became_active":"2024-08-31T22:12:48.393530+0000","last_became_peered":"2024-08-31T22:12:48.393530+0000","last_unstale":"2024-08-31T22:54:20.268404+0000","last_undegraded":"2024-08-31T22:54:20.268404+0000","last_fullsized":"2024-08-31T22:54:20.268404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:53:59.887015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021475320000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":688,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.270604+0000","last_change":"2024-08-31T22:12:50.394918+0000","last_active":"2024-08-31T22:54:20.270604+0000","last_peered":"2024-08-31T22:54:20.270604+0000","last_clean":"2024-08-31T22:54:20.270604+0000","last_became_active":"2024-08-31T22:12:48.390967+0000","last_became_peered":"2024-08-31T22:12:48.390967+0000","last_unstale":"2024-08-31T22:54:20.270604+0000","last_undegraded":"2024-08-31T22:54:20.270604+0000","last_fullsized":"2024-08-31T22:54:20.270604+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T07:18:29.001799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002301787,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":689,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.270653+0000","last_change":"2024-08-31T22:12:50.393340+0000","last_active":"2024-08-31T22:54:20.270653+0000","last_peered":"2024-08-31T22:54:20.270653+0000","last_clean":"2024-08-31T22:54:20.270653+0000","last_became_active":"2024-08-31T22:12:48.393810+0000","last_became_peered":"2024-08-31T22:12:48.393810+0000","last_unstale":"2024-08-31T22:54:20.270653+0000","last_undegraded":"2024-08-31T22:54:20.270653+0000","last_fullsized":"2024-08-31T22:54:20.270653+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:37:46.050434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011422590000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":689,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.268077+0000","last_change":"2024-08-31T22:12:50.394198+0000","last_active":"2024-08-31T22:54:20.268077+0000","last_peered":"2024-08-31T22:54:20.268077+0000","last_clean":"2024-08-31T22:54:20.268077+0000","last_became_active":"2024-08-31T22:12:48.395901+0000","last_became_peered":"2024-08-31T22:12:48.395901+0000","last_unstale":"2024-08-31T22:54:20.268077+0000","last_undegraded":"2024-08-31T22:54:20.268077+0000","last_fullsized":"2024-08-31T22:54:20.268077+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T02:29:23.840873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018848599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":690,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:18.977795+0000","last_change":"2024-08-31T22:12:50.400538+0000","last_active":"2024-08-31T22:54:18.977795+0000","last_peered":"2024-08-31T22:54:18.977795+0000","last_clean":"2024-08-31T22:54:18.977795+0000","last_became_active":"2024-08-31T22:12:48.389795+0000","last_became_peered":"2024-08-31T22:12:48.389795+0000","last_unstale":"2024-08-31T22:54:18.977795+0000","last_undegraded":"2024-08-31T22:54:18.977795+0000","last_fullsized":"2024-08-31T22:54:18.977795+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T01:35:29.294687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023100410000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":689,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.670977+0000","last_change":"2024-08-31T22:12:51.173642+0000","last_active":"2024-08-31T22:54:19.670977+0000","last_peered":"2024-08-31T22:54:19.670977+0000","last_clean":"2024-08-31T22:54:19.670977+0000","last_became_active":"2024-08-31T22:12:48.391597+0000","last_became_peered":"2024-08-31T22:12:48.391597+0000","last_unstale":"2024-08-31T22:54:19.670977+0000","last_undegraded":"2024-08-31T22:54:19.670977+0000","last_fullsized":"2024-08-31T22:54:19.670977+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:22:09.506410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051578099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":690,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.669859+0000","last_change":"2024-08-31T22:12:51.173571+0000","last_active":"2024-08-31T22:54:19.669859+0000","last_peered":"2024-08-31T22:54:19.669859+0000","last_clean":"2024-08-31T22:54:19.669859+0000","last_became_active":"2024-08-31T22:12:48.393856+0000","last_became_peered":"2024-08-31T22:12:48.393856+0000","last_unstale":"2024-08-31T22:54:19.669859+0000","last_undegraded":"2024-08-31T22:54:19.669859+0000","last_fullsized":"2024-08-31T22:54:19.669859+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:15:04.841041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046582699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":690,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.273404+0000","last_change":"2024-08-31T22:12:50.393581+0000","last_active":"2024-08-31T22:54:20.273404+0000","last_peered":"2024-08-31T22:54:20.273404+0000","last_clean":"2024-08-31T22:54:20.273404+0000","last_became_active":"2024-08-31T22:12:48.393780+0000","last_became_peered":"2024-08-31T22:12:48.393780+0000","last_unstale":"2024-08-31T22:54:20.273404+0000","last_undegraded":"2024-08-31T22:54:20.273404+0000","last_fullsized":"2024-08-31T22:54:20.273404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T01:27:52.811385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013191050000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":818,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.266663+0000","last_change":"2024-08-31T22:12:44.358595+0000","last_active":"2024-08-31T22:54:20.266663+0000","last_peered":"2024-08-31T22:54:20.266663+0000","last_clean":"2024-08-31T22:54:20.266663+0000","last_became_active":"2024-08-31T22:12:44.357729+0000","last_became_peered":"2024-08-31T22:12:44.357729+0000","last_unstale":"2024-08-31T22:54:20.266663+0000","last_undegraded":"2024-08-31T22:54:20.266663+0000","last_fullsized":"2024-08-31T22:54:20.266663+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:43.340061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:43.340061+0000","last_clean_scrub_stamp":"2024-08-31T22:12:43.340061+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:14:47.042415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6360,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706176,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1754900,"kb_used_data":2752,"kb_used_omap":124,"kb_used_meta":1752003,"kb_avail":92616940,"statfs":{"total":96636764160,"available":94839746560,"internally_reserved":0,"allocated":2818048,"data_stored":1481685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":127899,"internal_metadata":1794051173},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706177,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1439440,"kb_used_data":2812,"kb_used_omap":256,"kb_used_meta":1436351,"kb_avail":92932400,"statfs":{"total":96636764160,"available":95162777600,"internally_reserved":0,"allocated":2879488,"data_stored":2072072,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":263067,"internal_metadata":1470823525},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738882,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1550928,"kb_used_data":2812,"kb_used_omap":264,"kb_used_meta":1547831,"kb_avail":92820912,"statfs":{"total":96636764160,"available":95048613888,"internally_reserved":0,"allocated":2879488,"data_stored":2072072,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":270531,"internal_metadata":1584979773},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T22:55:10.553 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-08-31T22:55:10.554 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-08-31T22:55:10.845 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:55:10.846 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-08-31T22:55:10.859 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":1614,"stamp":"2024-08-31T22:55:09.237805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4745268,"kb_used_data":8376,"kb_used_omap":645,"kb_used_meta":4736186,"kb_avail":278370252,"statfs":{"total":289910292480,"available":285051138048,"internally_reserved":0,"allocated":8577024,"data_stored":5625829,"data_compressed":6360,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":661497,"internal_metadata":4849854471},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002987"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":688,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.268404+0000","last_change":"2024-08-31T22:12:50.394482+0000","last_active":"2024-08-31T22:54:20.268404+0000","last_peered":"2024-08-31T22:54:20.268404+0000","last_clean":"2024-08-31T22:54:20.268404+0000","last_became_active":"2024-08-31T22:12:48.393530+0000","last_became_peered":"2024-08-31T22:12:48.393530+0000","last_unstale":"2024-08-31T22:54:20.268404+0000","last_undegraded":"2024-08-31T22:54:20.268404+0000","last_fullsized":"2024-08-31T22:54:20.268404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:53:59.887015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021475320000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":688,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.270604+0000","last_change":"2024-08-31T22:12:50.394918+0000","last_active":"2024-08-31T22:54:20.270604+0000","last_peered":"2024-08-31T22:54:20.270604+0000","last_clean":"2024-08-31T22:54:20.270604+0000","last_became_active":"2024-08-31T22:12:48.390967+0000","last_became_peered":"2024-08-31T22:12:48.390967+0000","last_unstale":"2024-08-31T22:54:20.270604+0000","last_undegraded":"2024-08-31T22:54:20.270604+0000","last_fullsized":"2024-08-31T22:54:20.270604+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T07:18:29.001799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002301787,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":689,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.270653+0000","last_change":"2024-08-31T22:12:50.393340+0000","last_active":"2024-08-31T22:54:20.270653+0000","last_peered":"2024-08-31T22:54:20.270653+0000","last_clean":"2024-08-31T22:54:20.270653+0000","last_became_active":"2024-08-31T22:12:48.393810+0000","last_became_peered":"2024-08-31T22:12:48.393810+0000","last_unstale":"2024-08-31T22:54:20.270653+0000","last_undegraded":"2024-08-31T22:54:20.270653+0000","last_fullsized":"2024-08-31T22:54:20.270653+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:37:46.050434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011422590000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":689,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.268077+0000","last_change":"2024-08-31T22:12:50.394198+0000","last_active":"2024-08-31T22:54:20.268077+0000","last_peered":"2024-08-31T22:54:20.268077+0000","last_clean":"2024-08-31T22:54:20.268077+0000","last_became_active":"2024-08-31T22:12:48.395901+0000","last_became_peered":"2024-08-31T22:12:48.395901+0000","last_unstale":"2024-08-31T22:54:20.268077+0000","last_undegraded":"2024-08-31T22:54:20.268077+0000","last_fullsized":"2024-08-31T22:54:20.268077+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T02:29:23.840873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018848599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":690,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:18.977795+0000","last_change":"2024-08-31T22:12:50.400538+0000","last_active":"2024-08-31T22:54:18.977795+0000","last_peered":"2024-08-31T22:54:18.977795+0000","last_clean":"2024-08-31T22:54:18.977795+0000","last_became_active":"2024-08-31T22:12:48.389795+0000","last_became_peered":"2024-08-31T22:12:48.389795+0000","last_unstale":"2024-08-31T22:54:18.977795+0000","last_undegraded":"2024-08-31T22:54:18.977795+0000","last_fullsized":"2024-08-31T22:54:18.977795+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T01:35:29.294687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023100410000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":689,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.670977+0000","last_change":"2024-08-31T22:12:51.173642+0000","last_active":"2024-08-31T22:54:19.670977+0000","last_peered":"2024-08-31T22:54:19.670977+0000","last_clean":"2024-08-31T22:54:19.670977+0000","last_became_active":"2024-08-31T22:12:48.391597+0000","last_became_peered":"2024-08-31T22:12:48.391597+0000","last_unstale":"2024-08-31T22:54:19.670977+0000","last_undegraded":"2024-08-31T22:54:19.670977+0000","last_fullsized":"2024-08-31T22:54:19.670977+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:22:09.506410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051578099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":690,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.669859+0000","last_change":"2024-08-31T22:12:51.173571+0000","last_active":"2024-08-31T22:54:19.669859+0000","last_peered":"2024-08-31T22:54:19.669859+0000","last_clean":"2024-08-31T22:54:19.669859+0000","last_became_active":"2024-08-31T22:12:48.393856+0000","last_became_peered":"2024-08-31T22:12:48.393856+0000","last_unstale":"2024-08-31T22:54:19.669859+0000","last_undegraded":"2024-08-31T22:54:19.669859+0000","last_fullsized":"2024-08-31T22:54:19.669859+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:15:04.841041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046582699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":690,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.273404+0000","last_change":"2024-08-31T22:12:50.393581+0000","last_active":"2024-08-31T22:54:20.273404+0000","last_peered":"2024-08-31T22:54:20.273404+0000","last_clean":"2024-08-31T22:54:20.273404+0000","last_became_active":"2024-08-31T22:12:48.393780+0000","last_became_peered":"2024-08-31T22:12:48.393780+0000","last_unstale":"2024-08-31T22:54:20.273404+0000","last_undegraded":"2024-08-31T22:54:20.273404+0000","last_fullsized":"2024-08-31T22:54:20.273404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T01:27:52.811385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013191050000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":818,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.266663+0000","last_change":"2024-08-31T22:12:44.358595+0000","last_active":"2024-08-31T22:54:20.266663+0000","last_peered":"2024-08-31T22:54:20.266663+0000","last_clean":"2024-08-31T22:54:20.266663+0000","last_became_active":"2024-08-31T22:12:44.357729+0000","last_became_peered":"2024-08-31T22:12:44.357729+0000","last_unstale":"2024-08-31T22:54:20.266663+0000","last_undegraded":"2024-08-31T22:54:20.266663+0000","last_fullsized":"2024-08-31T22:54:20.266663+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:43.340061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:43.340061+0000","last_clean_scrub_stamp":"2024-08-31T22:12:43.340061+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:14:47.042415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6360,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706176,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1754900,"kb_used_data":2752,"kb_used_omap":124,"kb_used_meta":1752003,"kb_avail":92616940,"statfs":{"total":96636764160,"available":94839746560,"internally_reserved":0,"allocated":2818048,"data_stored":1481685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":127899,"internal_metadata":1794051173},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706177,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1439440,"kb_used_data":2812,"kb_used_omap":256,"kb_used_meta":1436351,"kb_avail":92932400,"statfs":{"total":96636764160,"available":95162777600,"internally_reserved":0,"allocated":2879488,"data_stored":2072072,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":263067,"internal_metadata":1470823525},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738882,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1550928,"kb_used_data":2812,"kb_used_omap":264,"kb_used_meta":1547831,"kb_avail":92820912,"statfs":{"total":96636764160,"available":95048613888,"internally_reserved":0,"allocated":2879488,"data_stored":2072072,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":270531,"internal_metadata":1584979773},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T22:55:10.860 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-08-31T22:55:11.157 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:55:11.157 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":342,"fsid":"8958e47a-d309-4bd5-ac60-bdf6f93061e5","created":"2024-08-31T22:12:31.498952+0000","modified":"2024-08-31T22:54:18.962806+0000","last_up_change":"2024-08-31T22:12:41.323775+0000","last_in_change":"2024-08-31T22:12:32.429991+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"octopus","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-31T22:12:42.384098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-31T22:12:46.929549+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7308cb5-1737-4816-a3fb-c3ed18553f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2244136370},{"type":"v1","addr":"172.21.15.7:6801","nonce":2244136370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2244136370},{"type":"v1","addr":"172.21.15.7:6803","nonce":2244136370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2244136370},{"type":"v1","addr":"172.21.15.7:6807","nonce":2244136370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2244136370},{"type":"v1","addr":"172.21.15.7:6805","nonce":2244136370}]},"public_addr":"172.21.15.7:6801/2244136370","cluster_addr":"172.21.15.7:6803/2244136370","heartbeat_back_addr":"172.21.15.7:6807/2244136370","heartbeat_front_addr":"172.21.15.7:6805/2244136370","state":["exists","up"]},{"osd":1,"uuid":"ce69f125-bfb1-463d-97f5-c2aa67f0124f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2889407906},{"type":"v1","addr":"172.21.15.7:6817","nonce":2889407906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":2889407906},{"type":"v1","addr":"172.21.15.7:6819","nonce":2889407906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":2889407906},{"type":"v1","addr":"172.21.15.7:6823","nonce":2889407906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":2889407906},{"type":"v1","addr":"172.21.15.7:6821","nonce":2889407906}]},"public_addr":"172.21.15.7:6817/2889407906","cluster_addr":"172.21.15.7:6819/2889407906","heartbeat_back_addr":"172.21.15.7:6823/2889407906","heartbeat_front_addr":"172.21.15.7:6821/2889407906","state":["exists","up"]},{"osd":2,"uuid":"f67f0693-4771-4a20-98a1-d2b6108a1ee0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2788741735},{"type":"v1","addr":"172.21.15.7:6809","nonce":2788741735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2788741735},{"type":"v1","addr":"172.21.15.7:6811","nonce":2788741735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2788741735},{"type":"v1","addr":"172.21.15.7:6815","nonce":2788741735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2788741735},{"type":"v1","addr":"172.21.15.7:6813","nonce":2788741735}]},"public_addr":"172.21.15.7:6809/2788741735","cluster_addr":"172.21.15.7:6811/2788741735","heartbeat_back_addr":"172.21.15.7:6815/2788741735","heartbeat_front_addr":"172.21.15.7:6813/2788741735","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:12:39.036879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:12:39.404760+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T22:12:39.188110+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T22:55:11.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 has been restored 2024-08-31T22:55:12.175 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-08-31T22:55:12.176 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-08-31T22:55:12.336 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-08-31T22:55:12.336 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-31T22:55:12.336 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-08-31T22:55:12.353 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-08-31T22:55:12.646 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 0 to deep-scrub 2024-08-31T22:55:12.666 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-08-31T22:55:12.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-08-31T22:55:12.821 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-08-31T22:55:12.822 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-31T22:55:12.822 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-08-31T22:55:12.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-08-31T22:55:13.135 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 1 to deep-scrub 2024-08-31T22:55:13.154 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-08-31T22:55:13.154 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-08-31T22:55:13.301 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-08-31T22:55:13.302 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-31T22:55:13.302 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-08-31T22:55:13.318 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-08-31T22:55:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 2 to deep-scrub 2024-08-31T22:55:13.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-08-31T22:55:13.924 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:55:13.924 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-08-31T22:55:13.943 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":1616,"stamp":"2024-08-31T22:55:13.238743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4745268,"kb_used_data":8376,"kb_used_omap":645,"kb_used_meta":4736186,"kb_avail":278370252,"statfs":{"total":289910292480,"available":285051138048,"internally_reserved":0,"allocated":8577024,"data_stored":5625829,"data_compressed":6360,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":661497,"internal_metadata":4849854471},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002905"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":688,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.268404+0000","last_change":"2024-08-31T22:12:50.394482+0000","last_active":"2024-08-31T22:54:20.268404+0000","last_peered":"2024-08-31T22:54:20.268404+0000","last_clean":"2024-08-31T22:54:20.268404+0000","last_became_active":"2024-08-31T22:12:48.393530+0000","last_became_peered":"2024-08-31T22:12:48.393530+0000","last_unstale":"2024-08-31T22:54:20.268404+0000","last_undegraded":"2024-08-31T22:54:20.268404+0000","last_fullsized":"2024-08-31T22:54:20.268404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:53:59.887015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021475320000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":688,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.270604+0000","last_change":"2024-08-31T22:12:50.394918+0000","last_active":"2024-08-31T22:54:20.270604+0000","last_peered":"2024-08-31T22:54:20.270604+0000","last_clean":"2024-08-31T22:54:20.270604+0000","last_became_active":"2024-08-31T22:12:48.390967+0000","last_became_peered":"2024-08-31T22:12:48.390967+0000","last_unstale":"2024-08-31T22:54:20.270604+0000","last_undegraded":"2024-08-31T22:54:20.270604+0000","last_fullsized":"2024-08-31T22:54:20.270604+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T07:18:29.001799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002301787,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":689,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.270653+0000","last_change":"2024-08-31T22:12:50.393340+0000","last_active":"2024-08-31T22:54:20.270653+0000","last_peered":"2024-08-31T22:54:20.270653+0000","last_clean":"2024-08-31T22:54:20.270653+0000","last_became_active":"2024-08-31T22:12:48.393810+0000","last_became_peered":"2024-08-31T22:12:48.393810+0000","last_unstale":"2024-08-31T22:54:20.270653+0000","last_undegraded":"2024-08-31T22:54:20.270653+0000","last_fullsized":"2024-08-31T22:54:20.270653+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:37:46.050434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011422590000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":689,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.268077+0000","last_change":"2024-08-31T22:12:50.394198+0000","last_active":"2024-08-31T22:54:20.268077+0000","last_peered":"2024-08-31T22:54:20.268077+0000","last_clean":"2024-08-31T22:54:20.268077+0000","last_became_active":"2024-08-31T22:12:48.395901+0000","last_became_peered":"2024-08-31T22:12:48.395901+0000","last_unstale":"2024-08-31T22:54:20.268077+0000","last_undegraded":"2024-08-31T22:54:20.268077+0000","last_fullsized":"2024-08-31T22:54:20.268077+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T02:29:23.840873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018848599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":691,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:32.783495+0000","last_change":"2024-08-31T22:12:50.400538+0000","last_active":"2024-08-31T22:54:32.783495+0000","last_peered":"2024-08-31T22:54:32.783495+0000","last_clean":"2024-08-31T22:54:32.783495+0000","last_became_active":"2024-08-31T22:12:48.389795+0000","last_became_peered":"2024-08-31T22:12:48.389795+0000","last_unstale":"2024-08-31T22:54:32.783495+0000","last_undegraded":"2024-08-31T22:54:32.783495+0000","last_fullsized":"2024-08-31T22:54:32.783495+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023100410000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":689,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.670977+0000","last_change":"2024-08-31T22:12:51.173642+0000","last_active":"2024-08-31T22:54:19.670977+0000","last_peered":"2024-08-31T22:54:19.670977+0000","last_clean":"2024-08-31T22:54:19.670977+0000","last_became_active":"2024-08-31T22:12:48.391597+0000","last_became_peered":"2024-08-31T22:12:48.391597+0000","last_unstale":"2024-08-31T22:54:19.670977+0000","last_undegraded":"2024-08-31T22:54:19.670977+0000","last_fullsized":"2024-08-31T22:54:19.670977+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:22:09.506410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051578099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":690,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:19.669859+0000","last_change":"2024-08-31T22:12:51.173571+0000","last_active":"2024-08-31T22:54:19.669859+0000","last_peered":"2024-08-31T22:54:19.669859+0000","last_clean":"2024-08-31T22:54:19.669859+0000","last_became_active":"2024-08-31T22:12:48.393856+0000","last_became_peered":"2024-08-31T22:12:48.393856+0000","last_unstale":"2024-08-31T22:54:19.669859+0000","last_undegraded":"2024-08-31T22:54:19.669859+0000","last_fullsized":"2024-08-31T22:54:19.669859+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:15:04.841041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046582699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":690,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.273404+0000","last_change":"2024-08-31T22:12:50.393581+0000","last_active":"2024-08-31T22:54:20.273404+0000","last_peered":"2024-08-31T22:54:20.273404+0000","last_clean":"2024-08-31T22:54:20.273404+0000","last_became_active":"2024-08-31T22:12:48.393780+0000","last_became_peered":"2024-08-31T22:12:48.393780+0000","last_unstale":"2024-08-31T22:54:20.273404+0000","last_undegraded":"2024-08-31T22:54:20.273404+0000","last_fullsized":"2024-08-31T22:54:20.273404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:47.363223+0000","last_clean_scrub_stamp":"2024-08-31T22:12:47.363223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T01:27:52.811385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013191050000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":818,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:54:20.266663+0000","last_change":"2024-08-31T22:12:44.358595+0000","last_active":"2024-08-31T22:54:20.266663+0000","last_peered":"2024-08-31T22:54:20.266663+0000","last_clean":"2024-08-31T22:54:20.266663+0000","last_became_active":"2024-08-31T22:12:44.357729+0000","last_became_peered":"2024-08-31T22:12:44.357729+0000","last_unstale":"2024-08-31T22:54:20.266663+0000","last_undegraded":"2024-08-31T22:54:20.266663+0000","last_fullsized":"2024-08-31T22:54:20.266663+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:12:43.340061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:12:43.340061+0000","last_clean_scrub_stamp":"2024-08-31T22:12:43.340061+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:14:47.042415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6360,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706177,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1754900,"kb_used_data":2752,"kb_used_omap":124,"kb_used_meta":1752003,"kb_avail":92616940,"statfs":{"total":96636764160,"available":94839746560,"internally_reserved":0,"allocated":2818048,"data_stored":1481685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":127899,"internal_metadata":1794051173},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706178,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1439440,"kb_used_data":2812,"kb_used_omap":256,"kb_used_meta":1436351,"kb_avail":92932400,"statfs":{"total":96636764160,"available":95162777600,"internally_reserved":0,"allocated":2879488,"data_stored":2072072,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":263067,"internal_metadata":1470823525},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738883,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1550928,"kb_used_data":2812,"kb_used_omap":264,"kb_used_meta":1547831,"kb_avail":92820912,"statfs":{"total":96636764160,"available":95048613888,"internally_reserved":0,"allocated":2879488,"data_stored":2072072,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":270531,"internal_metadata":1584979773},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T22:55:13.944 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-08-31T22:12:47.363223+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=12, tm_sec=47, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=55, tm_sec=11, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T22:55:13.944 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-08-31T22:12:47.363223+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=12, tm_sec=47, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=55, tm_sec=11, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T22:55:13.944 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-08-31T22:12:47.363223+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=12, tm_sec=47, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=55, tm_sec=11, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T22:55:13.944 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-08-31T22:12:47.363223+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=12, tm_sec=47, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=55, tm_sec=11, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T22:55:13.945 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-08-31T22:12:47.363223+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=12, tm_sec=47, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=55, tm_sec=11, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T22:55:13.945 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-08-31T22:12:47.363223+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=12, tm_sec=47, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=55, tm_sec=11, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T22:55:13.945 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-08-31T22:12:47.363223+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=12, tm_sec=47, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=55, tm_sec=11, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T22:55:13.945 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-08-31T22:12:47.363223+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=12, tm_sec=47, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=55, tm_sec=11, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T22:55:13.945 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-08-31T22:12:43.340061+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=12, tm_sec=43, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=22, tm_min=55, tm_sec=11, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T22:55:13.945 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-08-31T22:55:33.946 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-08-31T22:55:34.258 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-08-31T22:55:34.259 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-08-31T22:55:34.276 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":1626,"stamp":"2024-08-31T22:55:33.243593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4745256,"kb_used_data":8376,"kb_used_omap":645,"kb_used_meta":4736186,"kb_avail":278370264,"statfs":{"total":289910292480,"available":285051150336,"internally_reserved":0,"allocated":8577024,"data_stored":5625829,"data_compressed":6360,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":661497,"internal_metadata":4849854471},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002859"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":697,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:55:15.363473+0000","last_change":"2024-08-31T22:55:15.363526+0000","last_active":"2024-08-31T22:55:15.363473+0000","last_peered":"2024-08-31T22:55:15.363473+0000","last_clean":"2024-08-31T22:55:15.363473+0000","last_became_active":"2024-08-31T22:12:48.393530+0000","last_became_peered":"2024-08-31T22:12:48.393530+0000","last_unstale":"2024-08-31T22:55:15.363473+0000","last_undegraded":"2024-08-31T22:55:15.363473+0000","last_fullsized":"2024-08-31T22:55:15.363473+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:55:15.363272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:55:15.363272+0000","last_clean_scrub_stamp":"2024-08-31T22:55:15.363272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:59:21.619531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021475320000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":697,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:55:14.409391+0000","last_change":"2024-08-31T22:55:14.409446+0000","last_active":"2024-08-31T22:55:14.409391+0000","last_peered":"2024-08-31T22:55:14.409391+0000","last_clean":"2024-08-31T22:55:14.409391+0000","last_became_active":"2024-08-31T22:12:48.390967+0000","last_became_peered":"2024-08-31T22:12:48.390967+0000","last_unstale":"2024-08-31T22:55:14.409391+0000","last_undegraded":"2024-08-31T22:55:14.409391+0000","last_fullsized":"2024-08-31T22:55:14.409391+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:55:14.409193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:55:14.409193+0000","last_clean_scrub_stamp":"2024-08-31T22:55:14.409193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T08:18:30.263295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002301787,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":698,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:55:17.354668+0000","last_change":"2024-08-31T22:55:17.354725+0000","last_active":"2024-08-31T22:55:17.354668+0000","last_peered":"2024-08-31T22:55:17.354668+0000","last_clean":"2024-08-31T22:55:17.354668+0000","last_became_active":"2024-08-31T22:12:48.393810+0000","last_became_peered":"2024-08-31T22:12:48.393810+0000","last_unstale":"2024-08-31T22:55:17.354668+0000","last_undegraded":"2024-08-31T22:55:17.354668+0000","last_fullsized":"2024-08-31T22:55:17.354668+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:55:17.354474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:55:17.354474+0000","last_clean_scrub_stamp":"2024-08-31T22:55:17.354474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T09:29:37.886995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011422590000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":698,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:55:18.346206+0000","last_change":"2024-08-31T22:55:18.346262+0000","last_active":"2024-08-31T22:55:18.346206+0000","last_peered":"2024-08-31T22:55:18.346206+0000","last_clean":"2024-08-31T22:55:18.346206+0000","last_became_active":"2024-08-31T22:12:48.395901+0000","last_became_peered":"2024-08-31T22:12:48.395901+0000","last_unstale":"2024-08-31T22:55:18.346206+0000","last_undegraded":"2024-08-31T22:55:18.346206+0000","last_fullsized":"2024-08-31T22:55:18.346206+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:55:18.345980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:55:18.345980+0000","last_clean_scrub_stamp":"2024-08-31T22:55:18.345980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T06:23:00.453339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018848599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"342'3","reported_seq":700,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:55:13.578411+0000","last_change":"2024-08-31T22:55:13.578453+0000","last_active":"2024-08-31T22:55:13.578411+0000","last_peered":"2024-08-31T22:55:13.578411+0000","last_clean":"2024-08-31T22:55:13.578411+0000","last_became_active":"2024-08-31T22:12:48.389795+0000","last_became_peered":"2024-08-31T22:12:48.389795+0000","last_unstale":"2024-08-31T22:55:13.578411+0000","last_undegraded":"2024-08-31T22:55:13.578411+0000","last_fullsized":"2024-08-31T22:55:13.578411+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'3","last_scrub_stamp":"2024-08-31T22:55:13.578309+0000","last_deep_scrub":"342'3","last_deep_scrub_stamp":"2024-08-31T22:55:13.578309+0000","last_clean_scrub_stamp":"2024-08-31T22:55:13.578309+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T02:46:09.475157+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0023100410000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":698,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:55:13.670889+0000","last_change":"2024-08-31T22:55:13.670907+0000","last_active":"2024-08-31T22:55:13.670889+0000","last_peered":"2024-08-31T22:55:13.670889+0000","last_clean":"2024-08-31T22:55:13.670889+0000","last_became_active":"2024-08-31T22:12:48.391597+0000","last_became_peered":"2024-08-31T22:12:48.391597+0000","last_unstale":"2024-08-31T22:55:13.670889+0000","last_undegraded":"2024-08-31T22:55:13.670889+0000","last_fullsized":"2024-08-31T22:55:13.670889+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:55:13.670821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:55:13.670821+0000","last_clean_scrub_stamp":"2024-08-31T22:55:13.670821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T02:44:59.443307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00051578099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":699,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:55:14.688687+0000","last_change":"2024-08-31T22:55:14.688738+0000","last_active":"2024-08-31T22:55:14.688687+0000","last_peered":"2024-08-31T22:55:14.688687+0000","last_clean":"2024-08-31T22:55:14.688687+0000","last_became_active":"2024-08-31T22:12:48.393856+0000","last_became_peered":"2024-08-31T22:12:48.393856+0000","last_unstale":"2024-08-31T22:55:14.688687+0000","last_undegraded":"2024-08-31T22:55:14.688687+0000","last_fullsized":"2024-08-31T22:55:14.688687+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T22:55:14.688500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T22:55:14.688500+0000","last_clean_scrub_stamp":"2024-08-31T22:55:14.688500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T03:38:59.725393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046582699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"342'2","reported_seq":700,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:55:16.345676+0000","last_change":"2024-08-31T22:55:16.345738+0000","last_active":"2024-08-31T22:55:16.345676+0000","last_peered":"2024-08-31T22:55:16.345676+0000","last_clean":"2024-08-31T22:55:16.345676+0000","last_became_active":"2024-08-31T22:12:48.393780+0000","last_became_peered":"2024-08-31T22:12:48.393780+0000","last_unstale":"2024-08-31T22:55:16.345676+0000","last_undegraded":"2024-08-31T22:55:16.345676+0000","last_fullsized":"2024-08-31T22:55:16.345676+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'2","last_scrub_stamp":"2024-08-31T22:55:16.345490+0000","last_deep_scrub":"342'2","last_deep_scrub_stamp":"2024-08-31T22:55:16.345490+0000","last_clean_scrub_stamp":"2024-08-31T22:55:16.345490+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T05:10:25.899932+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013191050000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"342'99","reported_seq":829,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-08-31T22:55:13.412827+0000","last_change":"2024-08-31T22:55:13.412883+0000","last_active":"2024-08-31T22:55:13.412827+0000","last_peered":"2024-08-31T22:55:13.412827+0000","last_clean":"2024-08-31T22:55:13.412827+0000","last_became_active":"2024-08-31T22:12:44.357729+0000","last_became_peered":"2024-08-31T22:12:44.357729+0000","last_unstale":"2024-08-31T22:55:13.412827+0000","last_undegraded":"2024-08-31T22:55:13.412827+0000","last_fullsized":"2024-08-31T22:55:13.412827+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'99","last_scrub_stamp":"2024-08-31T22:55:13.412646+0000","last_deep_scrub":"342'99","last_deep_scrub_stamp":"2024-08-31T22:55:13.412646+0000","last_clean_scrub_stamp":"2024-08-31T22:55:13.412646+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T08:46:41.720977+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6360,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706181,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1754896,"kb_used_data":2752,"kb_used_omap":124,"kb_used_meta":1752003,"kb_avail":92616944,"statfs":{"total":96636764160,"available":94839750656,"internally_reserved":0,"allocated":2818048,"data_stored":1481685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":127899,"internal_metadata":1794051173},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706182,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1439436,"kb_used_data":2812,"kb_used_omap":256,"kb_used_meta":1436351,"kb_avail":92932404,"statfs":{"total":96636764160,"available":95162781696,"internally_reserved":0,"allocated":2879488,"data_stored":2072072,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":263067,"internal_metadata":1470823525},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738887,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1550924,"kb_used_data":2812,"kb_used_omap":264,"kb_used_meta":1547831,"kb_avail":92820916,"statfs":{"total":96636764160,"available":95048617984,"internally_reserved":0,"allocated":2879488,"data_stored":2072072,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":270531,"internal_metadata":1584979773},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3180,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T22:55:34.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-08-31T22:55:34.604 INFO:teuthology.misc:Shutting down mds daemons... 2024-08-31T22:55:34.604 INFO:teuthology.misc:Shutting down osd daemons... 2024-08-31T22:55:34.605 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-08-31T22:55:34.605 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T22:55:35.822 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 is failed for ~0s 2024-08-31T22:55:40.707 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-08-31T22:55:40.708 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-08-31T22:55:40.708 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T22:55:41.151 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 is failed for ~0s 2024-08-31T22:55:41.152 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 has been restored 2024-08-31T22:55:46.455 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 is failed for ~5s 2024-08-31T22:55:46.810 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-08-31T22:55:46.810 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-08-31T22:55:46.811 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T22:55:51.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 is failed for ~0s 2024-08-31T22:55:51.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 has been restored 2024-08-31T22:55:52.913 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-08-31T22:55:52.913 INFO:teuthology.misc:Shutting down mgr daemons... 2024-08-31T22:55:52.913 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-08-31T22:55:52.913 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T22:55:52.944 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-08-31T22:55:52.944 INFO:teuthology.misc:Shutting down mon daemons... 2024-08-31T22:55:52.945 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-08-31T22:55:52.945 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T22:55:53.003 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-08-31T22:55:53.003 INFO:tasks.ceph:Checking cluster log for badness... 2024-08-31T22:55:53.004 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-08-31T22:55:53.092 INFO:teuthology.orchestra.run.smithi007.stdout:2024-08-31T22:15:38.560806+0000 mon.a (mon.0) 240 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-08-31T22:55:53.092 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-08-31T22:55:53.092 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-08-31T22:55:53.148 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-08-31T22:55:53.204 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[WRN\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-08-31T22:55:53.259 INFO:teuthology.orchestra.run.smithi007.stdout:2024-08-31T22:15:38.560806+0000 mon.a (mon.0) 240 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-08-31T22:55:53.260 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi007.front.sepia.ceph.com 2024-08-31T22:55:53.260 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-08-31T22:55:53.793 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi007.front.sepia.ceph.com 2024-08-31T22:55:53.794 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-08-31T22:55:54.135 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi007.front.sepia.ceph.com 2024-08-31T22:55:54.136 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-08-31T22:55:54.454 INFO:tasks.ceph:Archiving mon data... 2024-08-31T22:55:54.454 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/teuthology-2024-08-31_21:16:02-rbd-squid-distro-default-smithi/7883608/data/cluster1.mon.a.tgz 2024-08-31T22:55:54.455 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-08-31T22:55:54.464 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.RXPnJ1mCMR 2024-08-31T22:55:54.465 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.RXPnJ1mCMR 2024-08-31T22:55:54.632 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.RXPnJ1mCMR 2024-08-31T22:55:54.723 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.RXPnJ1mCMR is 379KB 2024-08-31T22:55:54.784 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.RXPnJ1mCMR 2024-08-31T22:55:54.789 INFO:tasks.ceph:Cleaning ceph cluster... 2024-08-31T22:55:54.789 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-08-31T22:55:54.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-08-31T22:55:54.893 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-08-31T22:55:54.909 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-08-31T22:55:54.909 INFO:tasks.ceph:Archiving crash dumps... 2024-08-31T22:55:54.911 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-08-31_21:16:02-rbd-squid-distro-default-smithi/7883608/remote/smithi007/crash 2024-08-31T22:55:54.912 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-31T22:55:54.944 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-08-31_21:16:02-rbd-squid-distro-default-smithi/7883608/remote/smithi032/crash 2024-08-31T22:55:54.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-31T22:55:54.963 INFO:tasks.ceph:Compressing logs... 2024-08-31T22:55:54.963 DEBUG:teuthology.orchestra.run.smithi007:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-31T22:55:54.992 DEBUG:teuthology.orchestra.run.smithi032:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-31T22:55:55.011 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28346.log 2024-08-31T22:55:55.011 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28442.log 2024-08-31T22:55:55.011 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.28346.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28034.log 2024-08-31T22:55:55.011 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28346.log.gz 2024-08-31T22:55:55.011 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29045.log 2024-08-31T22:55:55.011 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.28442.log: gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-08-31T22:55:55.011 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.28034.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28442.log.gz 2024-08-31T22:55:55.012 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28034.log.gz 2024-08-31T22:55:55.012 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.29045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29019.log 2024-08-31T22:55:55.012 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29045.log.gz 2024-08-31T22:55:55.012 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-08-31T22:55:55.012 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28967.log 2024-08-31T22:55:55.012 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.29019.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29019.log.gz 2024-08-31T22:55:55.012 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25226.log 2024-08-31T22:55:55.012 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.28967.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.28967.log.gz 2024-08-31T22:55:55.012 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.29097.log 2024-08-31T22:55:55.023 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-08-31T22:55:55.023 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.25226.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25226.log.gz 2024-08-31T22:55:55.023 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.29097.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28156.log 2024-08-31T22:55:55.024 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29097.log.gz 2024-08-31T22:55:55.024 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21440.log 2024-08-31T22:55:55.024 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28104.log 2024-08-31T22:55:55.024 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.28156.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28156.log.gz 2024-08-31T22:55:55.024 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.tmp-client.admin.21440.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-08-31T22:55:55.024 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21440.log.gz 2024-08-31T22:55:55.024 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.28104.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.28104.log.gz 2024-08-31T22:55:55.024 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.28416.log 2024-08-31T22:55:55.024 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28189.log 2024-08-31T22:55:55.024 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-mgr.x.log: /var/log/ceph/cluster1-client.admin.28416.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-08-31T22:55:55.024 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28416.log.gz 2024-08-31T22:55:55.024 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.28189.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28189.log.gz 2024-08-31T22:55:55.024 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-08-31T22:55:55.025 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28188.log 2024-08-31T22:55:55.025 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-osd.2.log: /var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25292.log 2024-08-31T22:55:55.025 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.28188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29202.log 2024-08-31T22:55:55.025 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28188.log.gz 2024-08-31T22:55:55.025 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.25292.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25292.log.gzgzip 2024-08-31T22:55:55.025 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.28342.log 2024-08-31T22:55:55.025 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.29202.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29123.log 2024-08-31T22:55:55.025 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29202.log.gz 2024-08-31T22:55:55.025 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.28342.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28084.log 2024-08-31T22:55:55.025 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28342.log.gz 2024-08-31T22:55:55.025 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.29123.log: gzip 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29123.log.gz -5 2024-08-31T22:55:55.025 INFO:teuthology.orchestra.run.smithi007.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.28468.log 2024-08-31T22:55:55.025 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.28084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28338.log 2024-08-31T22:55:55.026 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.28468.log: 80.3% -- replaced with /var/log/ceph/cluster1-client.admin.28084.log.gz 2024-08-31T22:55:55.026 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25168.log 2024-08-31T22:55:55.026 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28468.log.gz 2024-08-31T22:55:55.026 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.28338.log: gzip -5 --verbose -- 0.0% /var/log/ceph/cluster1-client.admin.28941.log -- replaced with /var/log/ceph/cluster1-client.admin.28338.log.gz 2024-08-31T22:55:55.026 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-08-31T22:55:55.026 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.25168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27885.log 2024-08-31T22:55:55.026 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25168.log.gz 2024-08-31T22:55:55.026 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.28941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29228.log 2024-08-31T22:55:55.026 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28941.log.gz 2024-08-31T22:55:55.026 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.27885.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28267.log 2024-08-31T22:55:55.026 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27885.log.gz 2024-08-31T22:55:55.026 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.29228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28059.log 2024-08-31T22:55:55.027 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29228.log.gz 2024-08-31T22:55:55.027 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.28267.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29150.log 2024-08-31T22:55:55.027 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28267.log.gz 2024-08-31T22:55:55.027 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.28059.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29071.log 2024-08-31T22:55:55.027 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28059.log.gz 2024-08-31T22:55:55.027 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.29150.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29176.log 2024-08-31T22:55:55.027 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29150.log.gz 2024-08-31T22:55:55.027 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.29071.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28130.log 2024-08-31T22:55:55.027 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29071.log.gz 2024-08-31T22:55:55.027 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.29176.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28268.log 2024-08-31T22:55:55.027 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29176.log.gz 2024-08-31T22:55:55.027 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.28130.log: gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-08-31T22:55:55.027 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28130.log.gz 2024-08-31T22:55:55.028 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.28268.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25050.log 2024-08-31T22:55:55.028 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28268.log.gz 2024-08-31T22:55:55.028 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29252.log 2024-08-31T22:55:55.028 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.25050.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25144.log 2024-08-31T22:55:55.028 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25050.log.gz 2024-08-31T22:55:55.028 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.29252.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28190.log 2024-08-31T22:55:55.028 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29252.log.gz 2024-08-31T22:55:55.028 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25192.log 2024-08-31T22:55:55.028 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.25144.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25144.log.gz 2024-08-31T22:55:55.028 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.28190.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28190.log.gz 2024-08-31T22:55:55.028 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28993.log 2024-08-31T22:55:55.028 INFO:teuthology.orchestra.run.smithi007.stderr:gzip/var/log/ceph/cluster1-client.admin.25192.log: -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-08-31T22:55:55.028 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25192.log.gz 2024-08-31T22:55:55.029 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-client.admin.28993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28260.log 2024-08-31T22:55:55.029 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28993.log.gz 2024-08-31T22:55:55.029 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cluster1-osd.1.log: /var/log/ceph/cluster1-client.admin.28260.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28260.log.gz 2024-08-31T22:55:55.029 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/cluster1.log.gz 2024-08-31T22:55:55.032 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88175.log 2024-08-31T22:55:55.033 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59882.log 2024-08-31T22:55:55.033 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.88175.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86214.log 2024-08-31T22:55:55.033 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.59882.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56146.log 2024-08-31T22:55:55.033 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86214.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159068.log 2024-08-31T22:55:55.033 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.56146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106628.log 2024-08-31T22:55:55.033 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159068.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56146.log.gz 2024-08-31T22:55:55.033 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60103.log 2024-08-31T22:55:55.033 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100483.log 2024-08-31T22:55:55.033 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.106628.log: /var/log/ceph/cluster2-client.mirror.60103.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103977.log 2024-08-31T22:55:55.033 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.100483.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93290.log 2024-08-31T22:55:55.033 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.103977.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.159068.log.gz 2024-08-31T22:55:55.033 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40955.log 2024-08-31T22:55:55.033 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.93290.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47594.log 2024-08-31T22:55:55.034 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.40955.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29512.log 2024-08-31T22:55:55.034 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.47594.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60537.log 2024-08-31T22:55:55.034 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.29512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145181.log 2024-08-31T22:55:55.034 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60537.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100669.log 2024-08-31T22:55:55.034 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.145181.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130074.log 2024-08-31T22:55:55.034 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.100669.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163970.log 2024-08-31T22:55:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.130074.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160324.log 2024-08-31T22:55:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163970.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74308.log 2024-08-31T22:55:55.038 INFO:teuthology.orchestra.run.smithi032.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.145181.log.gz 2024-08-31T22:55:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.160324.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101475.log 2024-08-31T22:55:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.74308.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57266.log 2024-08-31T22:55:55.038 INFO:teuthology.orchestra.run.smithi032.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.163970.log.gz 2024-08-31T22:55:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101475.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52313.log 2024-08-31T22:55:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.57266.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42608.log 2024-08-31T22:55:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.52313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99735.log 2024-08-31T22:55:55.038 INFO:teuthology.orchestra.run.smithi032.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.160324.log.gz 2024-08-31T22:55:55.038 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57266.log.gz 2024-08-31T22:55:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.42608.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120412.log 2024-08-31T22:55:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.99735.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33797.log 2024-08-31T22:55:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.120412.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104012.log 2024-08-31T22:55:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33797.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38188.log 2024-08-31T22:55:55.039 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.88175.log.gz 2024-08-31T22:55:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.104012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136478.log 2024-08-31T22:55:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.38188.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97588.log 2024-08-31T22:55:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.136478.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72254.log 2024-08-31T22:55:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.97588.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106066.log 2024-08-31T22:55:55.048 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.72254.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69602.log 2024-08-31T22:55:55.049 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1%/var/log/ceph/cluster2-client.mirror.106066.log: -- replaced with /var/log/ceph/cluster2-client.mirror.60537.log.gz 2024-08-31T22:55:55.049 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103745.log 2024-08-31T22:55:55.049 INFO:teuthology.orchestra.run.smithi032.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136478.log.gz 2024-08-31T22:55:55.049 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38188.log.gz 2024-08-31T22:55:55.049 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.69602.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58797.log 2024-08-31T22:55:55.049 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.103745.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161626.log 2024-08-31T22:55:55.049 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58797.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65808.log 2024-08-31T22:55:55.049 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161626.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77188.log 2024-08-31T22:55:55.049 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.161626.log.gz 2024-08-31T22:55:55.049 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65808.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100111.log 2024-08-31T22:55:55.049 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77188.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157237.log 2024-08-31T22:55:55.049 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.100483.log.gz 2024-08-31T22:55:55.050 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.100111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114232.log 2024-08-31T22:55:55.050 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157237.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78180.log 2024-08-31T22:55:55.050 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.114232.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161266.log 2024-08-31T22:55:55.050 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.78180.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.157237.log.gzgzip 2024-08-31T22:55:55.050 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107713.log 2024-08-31T22:55:55.050 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161266.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91984.log 2024-08-31T22:55:55.050 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.161266.log.gz 2024-08-31T22:55:55.050 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.100669.log.gz 2024-08-31T22:55:55.050 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107713.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45596.log 2024-08-31T22:55:55.050 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.100111.log.gz 2024-08-31T22:55:55.050 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.91984.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.155012.log 2024-08-31T22:55:55.050 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.45596.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46474.log 2024-08-31T22:55:55.050 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.155012.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95592.log 2024-08-31T22:55:55.050 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.46474.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85687.log 2024-08-31T22:55:55.051 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.95592.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157896.log 2024-08-31T22:55:55.051 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.85687.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84160.log 2024-08-31T22:55:55.051 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59882.log.gz 2024-08-31T22:55:55.051 INFO:teuthology.orchestra.run.smithi032.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.155012.log.gz 2024-08-31T22:55:55.051 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157896.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137264.log 2024-08-31T22:55:55.051 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60103.log.gz 2024-08-31T22:55:55.051 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.84160.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.157896.log.gz 2024-08-31T22:55:55.051 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119235.log 2024-08-31T22:55:55.051 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.137264.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52503.log 2024-08-31T22:55:55.051 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73901.log 2024-08-31T22:55:55.051 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119235.log: /var/log/ceph/cluster2-client.mirror.52503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36940.log 2024-08-31T22:55:55.052 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.73901.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61281.log 2024-08-31T22:55:55.052 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.36940.log: gzip 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99735.log.gz 2024-08-31T22:55:55.052 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109616.log 2024-08-31T22:55:55.052 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61281.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36882.log 2024-08-31T22:55:55.063 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.109616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62750.log 2024-08-31T22:55:55.063 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.36882.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67734.log 2024-08-31T22:55:55.063 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.62750.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104442.log 2024-08-31T22:55:55.063 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67734.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165670.log 2024-08-31T22:55:55.063 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.104442.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119688.log 2024-08-31T22:55:55.064 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165670.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149720.log 2024-08-31T22:55:55.064 INFO:teuthology.orchestra.run.smithi032.stderr: 96.5% -- replaced with /var/log/ceph/cluster2-client.mirror.137264.log.gz 2024-08-31T22:55:55.064 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119688.log: 87.9%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.165670.log.gz -5 2024-08-31T22:55:55.064 INFO:teuthology.orchestra.run.smithi032.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.35913.log 2024-08-31T22:55:55.064 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.149720.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119791.log 2024-08-31T22:55:55.064 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.35913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86028.log 2024-08-31T22:55:55.064 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119791.log: 88.0% 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103977.log.gz 2024-08-31T22:55:55.064 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104012.log.gz 2024-08-31T22:55:55.064 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 -- replaced with /var/log/ceph/cluster1-client.mirror.149720.log.gz --verbose -- /var/log/ceph/cluster2-client.mirror.100204.log 2024-08-31T22:55:55.064 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:55.064 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86028.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103905.log 2024-08-31T22:55:55.064 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.100204.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55619.log 2024-08-31T22:55:55.064 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.103905.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44325.log 2024-08-31T22:55:55.065 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.55619.log: gzip 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106628.log.gz 2024-08-31T22:55:55.065 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121532.log 2024-08-31T22:55:55.065 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.44325.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164567.log 2024-08-31T22:55:55.065 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.121532.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163575.log 2024-08-31T22:55:55.065 INFO:teuthology.orchestra.run.smithi032.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121532.log.gz 2024-08-31T22:55:55.065 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164567.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162211.log 2024-08-31T22:55:55.065 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163575.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71099.log 2024-08-31T22:55:55.065 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162211.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108775.log 2024-08-31T22:55:55.066 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.162211.log.gz 2024-08-31T22:55:55.066 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69602.log.gz 2024-08-31T22:55:55.066 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.71099.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.166008.log 2024-08-31T22:55:55.066 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.108775.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94038.log 2024-08-31T22:55:55.066 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164173.log 2024-08-31T22:55:55.066 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.166008.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.166008.log.gz 2024-08-31T22:55:55.066 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94038.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49931.log 2024-08-31T22:55:55.066 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.164173.log: /var/log/ceph/cluster2-client.mirror.77901.log 2024-08-31T22:55:55.066 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49931.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44011.log 2024-08-31T22:55:55.066 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72254.log.gz/var/log/ceph/cluster2-client.mirror.77901.log: 2024-08-31T22:55:55.066 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112213.log 2024-08-31T22:55:55.066 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.44011.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161153.log 2024-08-31T22:55:55.066 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/cluster2-client.mirror.112213.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91573.log 2024-08-31T22:55:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161153.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.161153.log.gz 2024-08-31T22:55:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136500.log 2024-08-31T22:55:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.91573.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65808.log.gz 2024-08-31T22:55:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110647.log 2024-08-31T22:55:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.136500.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83916.log 2024-08-31T22:55:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.110647.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116896.log 2024-08-31T22:55:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83916.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74308.log.gz 2024-08-31T22:55:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157308.log 2024-08-31T22:55:55.068 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.116896.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120389.log 2024-08-31T22:55:55.068 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157308.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102595.log 2024-08-31T22:55:55.074 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.120389.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106442.log 2024-08-31T22:55:55.074 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45596.log.gz 2024-08-31T22:55:55.074 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.102595.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.157308.log.gz 2024-08-31T22:55:55.074 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.61281.log.gz 2024-08-31T22:55:55.074 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137197.log 2024-08-31T22:55:55.074 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144140.log 2024-08-31T22:55:55.074 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.106442.log: /var/log/ceph/cluster1-client.mirror.137197.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54371.log 2024-08-31T22:55:55.074 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.144140.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38653.log 2024-08-31T22:55:55.075 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137197.log.gz 2024-08-31T22:55:55.075 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58797.log.gz 88.0% 2024-08-31T22:55:55.075 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.144140.log.gz 2024-08-31T22:55:55.075 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151569.log 2024-08-31T22:55:55.075 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.54371.log: /var/log/ceph/cluster2-client.mirror.38653.log: 97.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60665.log 2024-08-31T22:55:55.075 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.136500.log.gz 2024-08-31T22:55:55.075 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.151569.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90470.log 2024-08-31T22:55:55.075 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60665.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76781.log 2024-08-31T22:55:55.075 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.90470.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101475.log.gz 2024-08-31T22:55:55.075 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.76781.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75847.log 2024-08-31T22:55:55.075 INFO:teuthology.orchestra.run.smithi032.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.151569.log.gz 2024-08-31T22:55:55.075 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121802.log 2024-08-31T22:55:55.075 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75847.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104760.log 2024-08-31T22:55:55.076 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.121802.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135536.log 2024-08-31T22:55:55.076 INFO:teuthology.orchestra.run.smithi032.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.121802.log.gz 2024-08-31T22:55:55.076 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44418.log 2024-08-31T22:55:55.076 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.119235.log.gz 2024-08-31T22:55:55.076 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.104760.log: /var/log/ceph/cluster2-client.mirror.135536.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106066.log.gz 2024-08-31T22:55:55.076 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106442.log.gz 2024-08-31T22:55:55.078 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41988.log 2024-08-31T22:55:55.078 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.44418.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90805.log 2024-08-31T22:55:55.078 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102723.log 2024-08-31T22:55:55.078 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.41988.log: /var/log/ceph/cluster2-client.mirror.90805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85970.log 2024-08-31T22:55:55.078 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138844.log 2024-08-31T22:55:55.078 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.102723.log: /var/log/ceph/cluster2-client.mirror.85970.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164699.log 2024-08-31T22:55:55.078 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.138844.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145700.log 2024-08-31T22:55:55.078 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1%/var/log/ceph/cluster2-client.mirror.164699.log: -- replaced with /var/log/ceph/cluster2-client.mirror.138844.log.gz 2024-08-31T22:55:55.078 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.104442.log.gz 2024-08-31T22:55:55.080 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161513.log 2024-08-31T22:55:55.080 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.145700.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.145700.log.gz 2024-08-31T22:55:55.080 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44883.log 2024-08-31T22:55:55.080 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161513.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.164699.log.gz 2024-08-31T22:55:55.080 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.161513.log.gz 2024-08-31T22:55:55.080 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94937.log 2024-08-31T22:55:55.081 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67734.log.gz 2024-08-31T22:55:55.081 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.44883.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71099.log.gz 2024-08-31T22:55:55.083 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.102595.log.gz 2024-08-31T22:55:55.085 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44325.log.gz 2024-08-31T22:55:55.085 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44418.log.gz 2024-08-31T22:55:55.086 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44883.log.gz 2024-08-31T22:55:55.086 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100204.log.gz 2024-08-31T22:55:55.090 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.84160.log.gz 2024-08-31T22:55:55.092 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.94038.log.gz 2024-08-31T22:55:55.092 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.110647.log.gz 2024-08-31T22:55:55.093 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83916.log.gz 2024-08-31T22:55:55.094 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73901.log.gz 2024-08-31T22:55:55.097 INFO:teuthology.orchestra.run.smithi032.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119688.log.gz 2024-08-31T22:55:55.098 INFO:teuthology.orchestra.run.smithi032.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.120389.log.gz 2024-08-31T22:55:55.100 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159168.log 2024-08-31T22:55:55.100 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38653.log.gz 2024-08-31T22:55:55.100 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94937.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28377.log 2024-08-31T22:55:55.100 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159168.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116675.log 2024-08-31T22:55:55.101 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28377.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148217.log 2024-08-31T22:55:55.101 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28377.log.gz 2024-08-31T22:55:55.102 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76688.log 2024-08-31T22:55:55.102 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.159168.log.gz 2024-08-31T22:55:55.102 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.116675.log: /var/log/ceph/cluster2-client.mirror.148217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88361.log 2024-08-31T22:55:55.102 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.76688.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150805.log 2024-08-31T22:55:55.103 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.88361.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163549.log 2024-08-31T22:55:55.103 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37161.log 2024-08-31T22:55:55.104 INFO:teuthology.orchestra.run.smithi032.stderr: 92.0%/var/log/ceph/cluster2-client.mirror.150805.log: -- replaced with /var/log/ceph/cluster2-client.mirror.148217.log.gz 2024-08-31T22:55:55.104 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163549.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.163549.log.gz 2024-08-31T22:55:55.104 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.37161.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104760.log.gz 2024-08-31T22:55:55.104 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156421.log 2024-08-31T22:55:55.105 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162323.log 2024-08-31T22:55:55.105 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156421.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73467.log 2024-08-31T22:55:55.106 INFO:teuthology.orchestra.run.smithi032.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.150805.log.gz 2024-08-31T22:55:55.106 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.156421.log.gz 2024-08-31T22:55:55.106 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162323.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116989.log 2024-08-31T22:55:55.106 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.162323.log.gz 2024-08-31T22:55:55.106 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.73467.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98766.log 2024-08-31T22:55:55.107 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.116989.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84753.log 2024-08-31T22:55:55.107 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76033.log 2024-08-31T22:55:55.107 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.98766.log: /var/log/ceph/cluster2-client.mirror.84753.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78777.log 2024-08-31T22:55:55.107 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54836.log 2024-08-31T22:55:55.108 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.76033.log: /var/log/ceph/cluster2-client.mirror.78777.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60010.log 2024-08-31T22:55:55.108 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.54836.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142418.log 2024-08-31T22:55:55.108 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138570.log 2024-08-31T22:55:55.108 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60010.log: /var/log/ceph/cluster2-client.mirror.142418.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63812.log 2024-08-31T22:55:55.109 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.138570.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58762.log 2024-08-31T22:55:55.110 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63812.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52313.log.gz 2024-08-31T22:55:55.110 INFO:teuthology.orchestra.run.smithi032.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138570.log.gz 2024-08-31T22:55:55.110 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52503.log.gz 2024-08-31T22:55:55.111 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110926.log 2024-08-31T22:55:55.111 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58762.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42829.log 2024-08-31T22:55:55.112 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.110926.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28447.log 2024-08-31T22:55:55.112 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.42829.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55619.log.gz 2024-08-31T22:55:55.113 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39556.log 2024-08-31T22:55:55.113 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28447.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28447.log.gz 2024-08-31T22:55:55.113 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113205.log 2024-08-31T22:55:55.114 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39556.log: 95.2% -- replaced with /var/log/ceph/cluster2-client.mirror.164567.log.gz 2024-08-31T22:55:55.114 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94468.log 2024-08-31T22:55:55.114 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.113205.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83319.log 2024-08-31T22:55:55.117 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94468.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160120.log 2024-08-31T22:55:55.117 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83319.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128320.log 2024-08-31T22:55:55.117 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.160120.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109616.log.gz 2024-08-31T22:55:55.118 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92542.log 2024-08-31T22:55:55.118 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.160120.log.gz 2024-08-31T22:55:55.118 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.128320.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.128320.log.gz 2024-08-31T22:55:55.119 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144550.log 2024-08-31T22:55:55.119 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.92542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29164.log 2024-08-31T22:55:55.120 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.144550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111523.log 2024-08-31T22:55:55.120 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.29164.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.144550.log.gz 2024-08-31T22:55:55.120 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77936.log 2024-08-31T22:55:55.121 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.111523.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147453.log 2024-08-31T22:55:55.121 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77936.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106965.log 2024-08-31T22:55:55.121 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97588.log.gz 2024-08-31T22:55:55.122 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.147453.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99886.log 2024-08-31T22:55:55.122 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.106965.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.107713.log.gz 2024-08-31T22:55:55.123 INFO:teuthology.orchestra.run.smithi032.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.147453.log.gz 2024-08-31T22:55:55.123 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91794.log 2024-08-31T22:55:55.124 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.99886.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102351.log 2024-08-31T22:55:55.124 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.91794.log: /var/log/ceph/cluster2-client.mirror.102351.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.108775.log.gz 2024-08-31T22:55:55.125 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.29503.log 2024-08-31T22:55:55.125 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108217.log 2024-08-31T22:55:55.126 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.0.29503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141515.log 2024-08-31T22:55:55.126 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.108217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91084.log 2024-08-31T22:55:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.141515.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85129.log 2024-08-31T22:55:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.91084.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70571.log 2024-08-31T22:55:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.85129.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.90470.log.gz 2024-08-31T22:55:55.127 INFO:teuthology.orchestra.run.smithi032.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141515.log.gz 2024-08-31T22:55:55.128 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105880.log 2024-08-31T22:55:55.129 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.70571.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45974.log 2024-08-31T22:55:55.129 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105880.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108589.log 2024-08-31T22:55:55.129 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.45974.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112213.log.gz 2024-08-31T22:55:55.130 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.108589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98894.log 2024-08-31T22:55:55.130 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93197.log 2024-08-31T22:55:55.131 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.98894.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77281.log 2024-08-31T22:55:55.131 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.93197.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.154155.log 2024-08-31T22:55:55.131 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62795.log 2024-08-31T22:55:55.132 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77281.log: /var/log/ceph/cluster2-client.mirror.154155.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114887.log 2024-08-31T22:55:55.132 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40955.log.gz 2024-08-31T22:55:55.133 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62795.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.154155.log.gz 2024-08-31T22:55:55.135 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57828.log 2024-08-31T22:55:55.135 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.114887.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76312.log 2024-08-31T22:55:55.135 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.85687.log.gz 2024-08-31T22:55:55.135 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.57828.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40021.log 2024-08-31T22:55:55.136 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.76312.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89144.log 2024-08-31T22:55:55.136 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86028.log.gz 2024-08-31T22:55:55.137 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.40021.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115962.log 2024-08-31T22:55:55.137 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86214.log.gz 2024-08-31T22:55:55.137 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89144.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144869.log 2024-08-31T22:55:55.138 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.115962.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37498.log 2024-08-31T22:55:55.138 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.144869.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.144869.log.gz 2024-08-31T22:55:55.138 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85970.log.gz 2024-08-31T22:55:55.139 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75847.log.gz 2024-08-31T22:55:55.139 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76033.log.gz 2024-08-31T22:55:55.141 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76688.log.gz 2024-08-31T22:55:55.141 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76781.log.gz 2024-08-31T22:55:55.143 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77060.log 2024-08-31T22:55:55.143 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.37498.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80796.log 2024-08-31T22:55:55.144 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77060.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149334.log 2024-08-31T22:55:55.144 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.80796.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40618.log 2024-08-31T22:55:55.145 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100262.log 2024-08-31T22:55:55.145 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.149334.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35913.log.gz 2024-08-31T22:55:55.145 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.40618.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.149334.log.gz 2024-08-31T22:55:55.145 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99235.log 2024-08-31T22:55:55.146 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.100262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164147.log 2024-08-31T22:55:55.146 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.99235.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88772.log 2024-08-31T22:55:55.146 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164147.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36882.log.gz 2024-08-31T22:55:55.147 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.164147.log.gz 2024-08-31T22:55:55.147 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36940.log.gz 2024-08-31T22:55:55.147 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.37161.log.gz 2024-08-31T22:55:55.148 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42329.log 2024-08-31T22:55:55.148 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.88772.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108833.log 2024-08-31T22:55:55.148 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.42329.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63812.log.gz 2024-08-31T22:55:55.149 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153024.log 2024-08-31T22:55:55.149 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.108833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57731.log 2024-08-31T22:55:55.149 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.153024.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93945.log 2024-08-31T22:55:55.149 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.57731.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.77188.log.gz 2024-08-31T22:55:55.150 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96805.log 2024-08-31T22:55:55.150 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.93945.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117732.log 2024-08-31T22:55:55.150 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96805.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.77901.log.gz 2024-08-31T22:55:55.151 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99328.log 2024-08-31T22:55:55.151 INFO:teuthology.orchestra.run.smithi032.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153024.log.gz 2024-08-31T22:55:55.151 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.117732.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52538.log 2024-08-31T22:55:55.151 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.99328.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145559.log 2024-08-31T22:55:55.152 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.52538.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71006.log 2024-08-31T22:55:55.152 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.145559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38281.log 2024-08-31T22:55:55.153 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.71006.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59448.log 2024-08-31T22:55:55.153 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.38281.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33797.log.gz 2024-08-31T22:55:55.153 INFO:teuthology.orchestra.run.smithi032.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.145559.log.gz 2024-08-31T22:55:55.154 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143248.log 2024-08-31T22:55:55.154 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.59448.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97309.log 2024-08-31T22:55:55.155 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32115.log 2024-08-31T22:55:55.155 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.143248.log: /var/log/ceph/cluster2-client.mirror.97309.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.143248.log.gz 2024-08-31T22:55:55.156 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28369.log 2024-08-31T22:55:55.156 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.32115.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.78180.log.gz 2024-08-31T22:55:55.156 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136166.log 2024-08-31T22:55:55.156 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28369.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115073.log 2024-08-31T22:55:55.156 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28369.log.gz 2024-08-31T22:55:55.157 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.136166.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.78777.log.gz 2024-08-31T22:55:55.157 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.115073.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136166.log.gz 2024-08-31T22:55:55.157 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162233.log 2024-08-31T22:55:55.157 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160613.log 2024-08-31T22:55:55.158 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162233.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108965.log 2024-08-31T22:55:55.159 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.160613.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.162233.log.gz 2024-08-31T22:55:55.159 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.160613.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76630.log 2024-08-31T22:55:55.160 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:55.160 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.108965.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67211.log 2024-08-31T22:55:55.160 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.76630.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80517.log 2024-08-31T22:55:55.160 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67211.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90584.log 2024-08-31T22:55:55.163 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.80517.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56239.log 2024-08-31T22:55:55.163 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.90584.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93910.log 2024-08-31T22:55:55.163 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.56239.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156905.log 2024-08-31T22:55:55.163 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.93910.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88737.log 2024-08-31T22:55:55.163 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156905.log: gzip -5 --verbose -- 88.3% /var/log/ceph/cluster2-client.mirror.102409.log 2024-08-31T22:55:55.163 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.44011.log.gz 2024-08-31T22:55:55.164 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.88737.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.156905.log.gz 2024-08-31T22:55:55.164 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46474.log.gz 2024-08-31T22:55:55.166 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.47594.log.gz 2024-08-31T22:55:55.170 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66238.log 2024-08-31T22:55:55.170 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37498.log.gz 2024-08-31T22:55:55.171 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.102409.log: -- /var/log/ceph/cluster2-client.mirror.69358.log 2024-08-31T22:55:55.171 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.66238.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98952.log 2024-08-31T22:55:55.171 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.69358.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38281.log.gz 2024-08-31T22:55:55.171 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.99235.log.gz 2024-08-31T22:55:55.171 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160658.log 2024-08-31T22:55:55.171 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100262.log.gz 2024-08-31T22:55:55.171 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31987.log 2024-08-31T22:55:55.171 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.98952.log: /var/log/ceph/cluster1-client.mirror.160658.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.160658.log.gz 2024-08-31T22:55:55.171 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41988.log.gz 2024-08-31T22:55:55.171 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118278.log 2024-08-31T22:55:55.171 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.31987.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53809.log 2024-08-31T22:55:55.176 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118278.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49931.log.gz 2024-08-31T22:55:55.176 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53809.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66928.log 2024-08-31T22:55:55.176 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93910.log.gz 2024-08-31T22:55:55.177 INFO:teuthology.orchestra.run.smithi032.stderr: /var/log/ceph/cluster2-client.mirror.48714.log 2024-08-31T22:55:55.177 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.66928.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52127.log 2024-08-31T22:55:55.177 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48714.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153299.log 2024-08-31T22:55:55.177 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.52127.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161311.log 2024-08-31T22:55:55.177 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.153299.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101417.log 2024-08-31T22:55:55.177 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161311.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.116896.log.gz 2024-08-31T22:55:55.177 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.153299.log.gz 2024-08-31T22:55:55.177 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.116989.log.gz 2024-08-31T22:55:55.177 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.161311.log.gz 2024-08-31T22:55:55.177 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57956.log 2024-08-31T22:55:55.177 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58762.log.gz 2024-08-31T22:55:55.177 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101417.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.60010.log.gz 2024-08-31T22:55:55.178 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66893.log 2024-08-31T22:55:55.179 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160928.log 2024-08-31T22:55:55.179 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.57956.log: /var/log/ceph/cluster2-client.mirror.66893.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50542.log 2024-08-31T22:55:55.179 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.160928.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162751.log 2024-08-31T22:55:55.180 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.160928.log.gz 2024-08-31T22:55:55.180 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50542.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42701.log 2024-08-31T22:55:55.180 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115331.log 2024-08-31T22:55:55.181 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.162751.log.gz 2024-08-31T22:55:55.181 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.42701.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97867.log 2024-08-31T22:55:55.181 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.115331.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69730.log 2024-08-31T22:55:55.182 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.97867.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86997.log 2024-08-31T22:55:55.182 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.69730.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.91984.log.gz 2024-08-31T22:55:55.183 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86997.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121619.log 2024-08-31T22:55:55.183 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94937.log.gz 2024-08-31T22:55:55.183 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59975.log 2024-08-31T22:55:55.183 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57022.log 2024-08-31T22:55:55.184 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.121619.log: /var/log/ceph/cluster2-client.mirror.59975.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116210.log 2024-08-31T22:55:55.184 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.95592.log.gz 2024-08-31T22:55:55.184 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.57022.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.77936.log.gz 2024-08-31T22:55:55.184 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104349.log 2024-08-31T22:55:55.185 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.116210.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138592.log 2024-08-31T22:55:55.186 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.104349.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40021.log.gz 2024-08-31T22:55:55.186 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.138592.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119298.log 2024-08-31T22:55:55.187 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71692.log 2024-08-31T22:55:55.187 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119298.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.98766.log.gz 2024-08-31T22:55:55.188 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.108589.log.gz 2024-08-31T22:55:55.191 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.93290.log.gz 2024-08-31T22:55:55.193 INFO:teuthology.orchestra.run.smithi007.stderr: 94.3% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-08-31T22:55:55.195 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.111523.log.gz 2024-08-31T22:55:55.195 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95251.log 2024-08-31T22:55:55.196 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.71692.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66556.log 2024-08-31T22:55:55.196 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.95251.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.114232.log.gz 2024-08-31T22:55:55.197 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111107.log 2024-08-31T22:55:55.197 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.66556.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54871.log 2024-08-31T22:55:55.199 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.111107.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40056.log 2024-08-31T22:55:55.199 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.54871.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42608.log.gz 2024-08-31T22:55:55.199 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42829.log.gz 2024-08-31T22:55:55.201 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42329.log.gz 2024-08-31T22:55:55.201 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108182.log 2024-08-31T22:55:55.201 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76630.log.gz 2024-08-31T22:55:55.201 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.40056.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79490.log 2024-08-31T22:55:55.203 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.108182.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75098.log 2024-08-31T22:55:55.203 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.79490.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45038.log 2024-08-31T22:55:55.204 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75098.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32770.log 2024-08-31T22:55:55.204 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.45038.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50670.log 2024-08-31T22:55:55.205 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.83319.log.gz 2024-08-31T22:55:55.206 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.32770.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85559.log 2024-08-31T22:55:55.206 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50670.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33049.log 2024-08-31T22:55:55.206 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.85559.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.91573.log.gz 2024-08-31T22:55:55.208 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.91794.log.gz 2024-08-31T22:55:55.208 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149765.log 2024-08-31T22:55:55.208 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33049.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158207.log 2024-08-31T22:55:55.209 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.149765.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.95251.log.gz 2024-08-31T22:55:55.210 INFO:teuthology.orchestra.run.smithi032.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.149765.log.gz 2024-08-31T22:55:55.212 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84908.log 2024-08-31T22:55:55.212 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158207.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.158207.log.gz 2024-08-31T22:55:55.214 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43821.log 2024-08-31T22:55:55.214 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.84908.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25347.log 2024-08-31T22:55:55.214 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.43821.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.92542.log.gz 2024-08-31T22:55:55.215 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160446.log 2024-08-31T22:55:55.216 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.25347.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.93197.log.gz 2024-08-31T22:55:55.216 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25347.log.gz 2024-08-31T22:55:55.216 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97867.log.gz 2024-08-31T22:55:55.219 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.80796.log.gz 2024-08-31T22:55:55.220 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96654.log 2024-08-31T22:55:55.221 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.160446.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.160446.log.gz 2024-08-31T22:55:55.221 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.98952.log.gz 2024-08-31T22:55:55.221 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57022.log.gz 2024-08-31T22:55:55.222 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57731.log.gz 2024-08-31T22:55:55.223 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84346.log 2024-08-31T22:55:55.223 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96654.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32208.log 2024-08-31T22:55:55.224 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.84346.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76909.log 2024-08-31T22:55:55.225 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.32208.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.77060.log.gz 2024-08-31T22:55:55.225 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48904.log 2024-08-31T22:55:55.225 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.76909.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102971.log 2024-08-31T22:55:55.226 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48904.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141326.log 2024-08-31T22:55:55.227 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.102971.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161716.log 2024-08-31T22:55:55.227 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.141326.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141326.log.gz 2024-08-31T22:55:55.228 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87617.log 2024-08-31T22:55:55.229 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161716.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.161716.log.gz 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86997.log.gz 2024-08-31T22:55:55.229 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:55.229 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74215.log 2024-08-31T22:55:55.229 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.87617.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124574.log 2024-08-31T22:55:55.230 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.74215.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86718.log 2024-08-31T22:55:55.231 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.124574.log: 95.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130074.log.gz 2024-08-31T22:55:55.232 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78649.log 2024-08-31T22:55:55.232 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32115.log.gz 2024-08-31T22:55:55.232 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86718.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59541.log 2024-08-31T22:55:55.233 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.78649.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67548.log 2024-08-31T22:55:55.234 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.59541.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159949.log 2024-08-31T22:55:55.235 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67548.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59262.log 2024-08-31T22:55:55.237 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159949.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.159949.log.gz 2024-08-31T22:55:55.239 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88830.log 2024-08-31T22:55:55.239 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.59262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56088.log 2024-08-31T22:55:55.240 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.88830.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43205.log 2024-08-31T22:55:55.241 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.56088.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57638.log 2024-08-31T22:55:55.242 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.43205.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101324.log 2024-08-31T22:55:55.243 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.57638.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67397.log 2024-08-31T22:55:55.244 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101324.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159261.log 2024-08-31T22:55:55.244 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67397.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153712.log 2024-08-31T22:55:55.245 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159261.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.159261.log.gz 2024-08-31T22:55:55.246 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68761.log 2024-08-31T22:55:55.247 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.153712.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.153712.log.gz 2024-08-31T22:55:55.247 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113895.log 2024-08-31T22:55:55.249 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.68761.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71006.log.gz 2024-08-31T22:55:55.249 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163783.log 2024-08-31T22:55:55.251 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.108965.log.gz/var/log/ceph/cluster2-client.mirror.113895.log: 2024-08-31T22:55:55.251 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73432.log 2024-08-31T22:55:55.251 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101417.log.gz 2024-08-31T22:55:55.252 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71692.log.gz 2024-08-31T22:55:55.252 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163783.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.96654.log.gz 2024-08-31T22:55:55.252 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.102409.log.gz 2024-08-31T22:55:55.252 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.119298.log.gz 2024-08-31T22:55:55.252 INFO:teuthology.orchestra.run.smithi032.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.163783.log.gz 2024-08-31T22:55:55.252 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75506.log 2024-08-31T22:55:55.253 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.73432.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148889.log 2024-08-31T22:55:55.254 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75506.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89950.log 2024-08-31T22:55:55.255 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.148889.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.148889.log.gz 2024-08-31T22:55:55.255 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45724.log 2024-08-31T22:55:55.256 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89950.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40618.log.gz 2024-08-31T22:55:55.256 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117774.log 2024-08-31T22:55:55.257 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93945.log.gz 2024-08-31T22:55:55.258 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.45724.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50340.log 2024-08-31T22:55:55.258 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.117774.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93011.log 2024-08-31T22:55:55.259 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50340.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96805.log.gz 2024-08-31T22:55:55.260 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71413.log 2024-08-31T22:55:55.260 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.93011.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97309.log.gz 2024-08-31T22:55:55.260 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59448.log.gz 2024-08-31T22:55:55.263 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99421.log 2024-08-31T22:55:55.263 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.71413.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118638.log 2024-08-31T22:55:55.264 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.99421.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112178.log 2024-08-31T22:55:55.264 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.114887.log.gz 2024-08-31T22:55:55.265 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118638.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81265.log 2024-08-31T22:55:55.266 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.112178.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37874.log 2024-08-31T22:55:55.267 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.81265.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46532.log 2024-08-31T22:55:55.267 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.37874.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140940.log 2024-08-31T22:55:55.268 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.46532.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31429.log 2024-08-31T22:55:55.269 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.140940.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.140940.log.gz 2024-08-31T22:55:55.270 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78870.log 2024-08-31T22:55:55.270 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.31429.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60723.log 2024-08-31T22:55:55.271 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.78870.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161378.log 2024-08-31T22:55:55.271 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.115962.log.gz 2024-08-31T22:55:55.272 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60723.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116210.log.gz 2024-08-31T22:55:55.273 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.116675.log.gz 2024-08-31T22:55:55.273 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45974.log.gz 2024-08-31T22:55:55.273 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91161.log 2024-08-31T22:55:55.274 INFO:teuthology.orchestra.run.smithi032.stderr: 63.7% -- replaced with /var/log/ceph/cluster1-client.mirror.29164.log.gz 2024-08-31T22:55:55.275 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161378.log: 86.8% 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.161378.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.56088.log.gz 2024-08-31T22:55:55.275 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:55.275 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111060.log 2024-08-31T22:55:55.275 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.91161.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56239.log.gz 2024-08-31T22:55:55.276 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.94468.log.gz 2024-08-31T22:55:55.277 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57828.log.gz 2024-08-31T22:55:55.279 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67211.log.gz 2024-08-31T22:55:55.280 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60665.log.gz 2024-08-31T22:55:55.281 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77281.log.gz 2024-08-31T22:55:55.282 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.54371.log.gz 2024-08-31T22:55:55.283 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65587.log 2024-08-31T22:55:55.283 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54836.log.gz 2024-08-31T22:55:55.283 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.111060.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54871.log.gz 2024-08-31T22:55:55.284 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73657.log 2024-08-31T22:55:55.284 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40056.log.gz 2024-08-31T22:55:55.284 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65587.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103099.log 2024-08-31T22:55:55.285 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.79490.log.gz 2024-08-31T22:55:55.285 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.73657.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108310.log 2024-08-31T22:55:55.286 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.103099.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47187.log 2024-08-31T22:55:55.286 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.108310.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160537.log 2024-08-31T22:55:55.287 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.88361.log.gz 2024-08-31T22:55:55.288 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.47187.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.88737.log.gz 2024-08-31T22:55:55.288 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.90805.log.gz 2024-08-31T22:55:55.288 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.88772.log.gz 2024-08-31T22:55:55.289 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.91084.log.gz 2024-08-31T22:55:55.290 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.108217.log.gz 89.0% 2024-08-31T22:55:55.290 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.108182.log.gz 2024-08-31T22:55:55.290 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.112178.log.gz 2024-08-31T22:55:55.290 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119711.log 2024-08-31T22:55:55.291 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.160537.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102351.log.gz 2024-08-31T22:55:55.291 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.102723.log.gz 2024-08-31T22:55:55.292 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42701.log.gz 2024-08-31T22:55:55.293 INFO:teuthology.orchestra.run.smithi032.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.160537.log.gz 2024-08-31T22:55:55.293 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110143.log 2024-08-31T22:55:55.293 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93011.log.gz 2024-08-31T22:55:55.294 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119711.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73467.log.gz 2024-08-31T22:55:55.294 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.80517.log.gz 2024-08-31T22:55:55.296 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57638.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.66928.log.gz 2024-08-31T22:55:55.296 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:55.296 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66893.log.gz 2024-08-31T22:55:55.297 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99084.log 2024-08-31T22:55:55.297 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.110143.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69358.log.gz 2024-08-31T22:55:55.297 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.86718.log.gz 2024-08-31T22:55:55.299 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85466.log 2024-08-31T22:55:55.299 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45724.log.gz 2024-08-31T22:55:55.300 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.99084.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90438.log 2024-08-31T22:55:55.300 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.84753.log.gz 2024-08-31T22:55:55.300 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.85466.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65587.log.gz 2024-08-31T22:55:55.301 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66052.log 2024-08-31T22:55:55.301 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.90438.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67548.log.gz 2024-08-31T22:55:55.301 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73432.log.gz 2024-08-31T22:55:55.301 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74215.log.gz 2024-08-31T22:55:55.302 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114422.log 2024-08-31T22:55:55.302 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.66052.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46753.log 2024-08-31T22:55:55.303 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66238.log.gz 2024-08-31T22:55:55.303 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.114422.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39963.log 2024-08-31T22:55:55.304 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.46753.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155844.log 2024-08-31T22:55:55.304 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39963.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67397.log.gz 2024-08-31T22:55:55.305 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88082.log 2024-08-31T22:55:55.306 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.155844.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155844.log.gz 2024-08-31T22:55:55.306 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38715.log 2024-08-31T22:55:55.306 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71413.log.gz 2024-08-31T22:55:55.307 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.88082.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37874.log.gz 2024-08-31T22:55:55.307 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162256.log 2024-08-31T22:55:55.308 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.38715.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124551.log 2024-08-31T22:55:55.309 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162256.log: 86.8% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39556.log.gz 2024-08-31T22:55:55.309 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.162256.log.gz 2024-08-31T22:55:55.309 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99328.log.gz 2024-08-31T22:55:55.309 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50542.log.gz 2024-08-31T22:55:55.309 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.59541.log.gz 2024-08-31T22:55:55.309 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.admin.115331.log.gz 2024-08-31T22:55:55.310 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59975.log.gz 2024-08-31T22:55:55.310 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113333.log 2024-08-31T22:55:55.311 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.124551.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.124551.log.gz 2024-08-31T22:55:55.311 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51662.log 2024-08-31T22:55:55.312 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.113333.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28143.log 2024-08-31T22:55:55.313 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51662.log: gzip 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.106965.log.gz -5 2024-08-31T22:55:55.313 INFO:teuthology.orchestra.run.smithi032.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.60382.log 2024-08-31T22:55:55.313 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28143.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28143.log.gz 2024-08-31T22:55:55.313 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161131.log 2024-08-31T22:55:55.314 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69730.log.gz 2024-08-31T22:55:55.315 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60382.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.162882.log 2024-08-31T22:55:55.315 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161131.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75506.log.gz 2024-08-31T22:55:55.315 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46753.log.gz 2024-08-31T22:55:55.316 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.161131.log.gz 2024-08-31T22:55:55.317 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86621.log 2024-08-31T22:55:55.317 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.162882.log: 55.3% 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.47187.log.gz 2024-08-31T22:55:55.317 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.162882.log.gz 2024-08-31T22:55:55.318 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60630.log 2024-08-31T22:55:55.318 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110926.log.gz 2024-08-31T22:55:55.318 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86621.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61878.log 2024-08-31T22:55:55.318 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60630.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52352.log 2024-08-31T22:55:55.319 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61878.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71285.log 2024-08-31T22:55:55.320 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.52352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92635.log 2024-08-31T22:55:55.321 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.71285.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87745.log 2024-08-31T22:55:55.321 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.92635.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.113205.log.gz 2024-08-31T22:55:55.323 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74928.log 2024-08-31T22:55:55.324 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.87745.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99886.log.gz 2024-08-31T22:55:55.325 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30402.log 2024-08-31T22:55:55.325 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.74928.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39401.log 2024-08-31T22:55:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.30402.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105694.log 2024-08-31T22:55:55.327 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39401.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91666.log 2024-08-31T22:55:55.327 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105694.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121245.log 2024-08-31T22:55:55.327 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.91666.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52538.log.gz 2024-08-31T22:55:55.328 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52352.log.gz 2024-08-31T22:55:55.330 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.121245.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93759.log 2024-08-31T22:55:55.330 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75098.log.gz 2024-08-31T22:55:55.330 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38715.log.gz 2024-08-31T22:55:55.333 INFO:teuthology.orchestra.run.smithi032.stderr: 19.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.29503.log.gz 2024-08-31T22:55:55.335 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.78649.log.gz 2024-08-31T22:55:55.335 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.110143.log.gz 2024-08-31T22:55:55.335 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.78870.log.gz 2024-08-31T22:55:55.335 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.90438.log.gz 2024-08-31T22:55:55.339 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99084.log.gz 2024-08-31T22:55:55.342 INFO:teuthology.orchestra.run.smithi032.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.164173.log.gz 2024-08-31T22:55:55.343 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41083.log 2024-08-31T22:55:55.344 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.93759.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150159.log 2024-08-31T22:55:55.344 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.41083.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25195.log 2024-08-31T22:55:55.345 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.150159.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.113895.log.gz 2024-08-31T22:55:55.346 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.150159.log.gz 2024-08-31T22:55:55.346 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158875.log 2024-08-31T22:55:55.347 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.25195.log: 0.0% 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.114422.log.gz 2024-08-31T22:55:55.347 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.25195.log.gz 2024-08-31T22:55:55.347 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108868.log 2024-08-31T22:55:55.348 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158875.log: 89.1% 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.115073.log.gz 2024-08-31T22:55:55.348 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.158875.log.gz 2024-08-31T22:55:55.349 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76312.log.gz 2024-08-31T22:55:55.349 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148637.log 2024-08-31T22:55:55.349 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.92635.log.gz 2024-08-31T22:55:55.350 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.108868.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62703.log 2024-08-31T22:55:55.350 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.148637.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.91666.log.gz 2024-08-31T22:55:55.350 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.86621.log.gz 2024-08-31T22:55:55.352 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48904.log.gz 2024-08-31T22:55:55.354 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43821.log.gz 2024-08-31T22:55:55.354 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88082.log.gz 2024-08-31T22:55:55.358 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57956.log.gz 2024-08-31T22:55:55.359 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66052.log.gz 2024-08-31T22:55:55.359 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.135536.log.gz 2024-08-31T22:55:55.360 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.48714.log.gz 2024-08-31T22:55:55.360 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60382.log.gz 2024-08-31T22:55:55.363 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51662.log.gz 2024-08-31T22:55:55.367 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.84908.log.gz 2024-08-31T22:55:55.368 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85129.log.gz 2024-08-31T22:55:55.368 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.85466.log.gz 2024-08-31T22:55:55.368 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85559.log.gz 2024-08-31T22:55:55.369 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71285.log.gz 2024-08-31T22:55:55.370 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73657.log.gz 2024-08-31T22:55:55.371 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74928.log.gz 2024-08-31T22:55:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160973.log 2024-08-31T22:55:55.371 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.87617.log.gz 2024-08-31T22:55:55.372 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.62703.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.104349.log.gz 2024-08-31T22:55:55.372 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163446.log 2024-08-31T22:55:55.372 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.160973.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.160973.log.gz 2024-08-31T22:55:55.373 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39963.log.gz 2024-08-31T22:55:55.373 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50670.log.gz 2024-08-31T22:55:55.373 INFO:teuthology.orchestra.run.smithi032.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103745.log.gz 2024-08-31T22:55:55.374 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117516.log 2024-08-31T22:55:55.374 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163446.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.163446.log.gz 2024-08-31T22:55:55.375 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71378.log 2024-08-31T22:55:55.376 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.117516.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68482.log 2024-08-31T22:55:55.380 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.71378.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46532.log.gz 2024-08-31T22:55:55.381 INFO:teuthology.orchestra.run.smithi032.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.148637.log.gz 2024-08-31T22:55:55.382 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162008.log 2024-08-31T22:55:55.382 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61037.log 2024-08-31T22:55:55.382 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.68482.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32208.log.gz 2024-08-31T22:55:55.383 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162008.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161468.log 2024-08-31T22:55:55.384 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.162008.log.gz 2024-08-31T22:55:55.384 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61037.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32770.log.gz 2024-08-31T22:55:55.384 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44980.log 2024-08-31T22:55:55.384 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161468.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33049.log.gz 2024-08-31T22:55:55.384 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.161468.log.gz 2024-08-31T22:55:55.385 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61223.log 2024-08-31T22:55:55.385 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.44980.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89578.log 2024-08-31T22:55:55.385 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.154197.log 2024-08-31T22:55:55.386 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61223.log: /var/log/ceph/cluster2-client.mirror.89578.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71192.log 2024-08-31T22:55:55.386 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.89144.log.gz 2024-08-31T22:55:55.386 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29301.log 2024-08-31T22:55:55.387 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.154197.log: /var/log/ceph/cluster2-client.mirror.71192.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105787.log 2024-08-31T22:55:55.387 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.29301.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153334.log 2024-08-31T22:55:55.388 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105787.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104853.log 2024-08-31T22:55:55.389 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.153334.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.153334.log.gz 2024-08-31T22:55:55.389 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36471.log 2024-08-31T22:55:55.389 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.104853.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69544.log 2024-08-31T22:55:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.36471.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140975.log 2024-08-31T22:55:55.394 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.69544.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149023.log 2024-08-31T22:55:55.394 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.140975.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.140975.log.gz 2024-08-31T22:55:55.396 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57080.log 2024-08-31T22:55:55.399 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.149023.log: 96.9% -- replaced with /var/log/ceph/cluster2-client.mirror.154197.log.gz 2024-08-31T22:55:55.400 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.163721.log 2024-08-31T22:55:55.400 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.57080.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108027.log 2024-08-31T22:55:55.401 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.163721.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.163721.log.gz 2024-08-31T22:55:55.402 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34638.log 2024-08-31T22:55:55.402 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.108027.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161963.log 2024-08-31T22:55:55.403 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.34638.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81672.log 2024-08-31T22:55:55.405 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161963.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.161963.log.gz 2024-08-31T22:55:55.405 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121222.log 2024-08-31T22:55:55.406 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.81672.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54743.log 2024-08-31T22:55:55.407 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.121222.log: 82.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121222.log.gz 2024-08-31T22:55:55.407 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90607.log 2024-08-31T22:55:55.408 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.54743.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118818.log 2024-08-31T22:55:55.409 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.90607.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63626.log 2024-08-31T22:55:55.409 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118818.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52817.log 2024-08-31T22:55:55.412 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28724.log 2024-08-31T22:55:55.412 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.52817.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41083.log.gz 2024-08-31T22:55:55.412 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112306.log 2024-08-31T22:55:55.412 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.28724.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28724.log.gz 2024-08-31T22:55:55.413 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114457.log 2024-08-31T22:55:55.414 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.112306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135892.log 2024-08-31T22:55:55.414 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.114457.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49555.log 2024-08-31T22:55:55.416 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.135892.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135892.log.gz 2024-08-31T22:55:55.416 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31557.log 2024-08-31T22:55:55.417 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49555.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69323.log 2024-08-31T22:55:55.418 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.31557.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.93759.log.gz 2024-08-31T22:55:55.418 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52127.log.gz 2024-08-31T22:55:55.419 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.108310.log.gz 2024-08-31T22:55:55.420 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.108833.log.gz 2024-08-31T22:55:55.420 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.108868.log.gz 2024-08-31T22:55:55.423 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70571.log.gz 2024-08-31T22:55:55.424 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50271.log 2024-08-31T22:55:55.425 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.69323.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107188.log 2024-08-31T22:55:55.425 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.50271.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50914.log 2024-08-31T22:55:55.426 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107188.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101231.log 2024-08-31T22:55:55.426 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47559.log 2024-08-31T22:55:55.427 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101231.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31053.log 2024-08-31T22:55:55.428 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.47559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130481.log 2024-08-31T22:55:55.429 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.31053.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76909.log.gz 2024-08-31T22:55:55.429 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34417.log 2024-08-31T22:55:55.435 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.130481.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.98894.log.gz 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57080.log.gz 2024-08-31T22:55:55.435 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:55.437 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63626.log.gz 2024-08-31T22:55:55.437 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.54743.log.gz 2024-08-31T22:55:55.440 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31987.log.gz 2024-08-31T22:55:55.443 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70199.log 2024-08-31T22:55:55.443 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30402.log.gz 2024-08-31T22:55:55.446 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.34417.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87745.log.gz 2024-08-31T22:55:55.449 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81265.log.gz 2024-08-31T22:55:55.451 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.89950.log.gz 2024-08-31T22:55:55.454 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71192.log.gz 2024-08-31T22:55:55.455 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71378.log.gz 2024-08-31T22:55:55.455 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101324.log.gz 2024-08-31T22:55:55.455 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.138592.log.gz 2024-08-31T22:55:55.455 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84943.log 2024-08-31T22:55:55.456 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.70199.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160951.log 2024-08-31T22:55:55.456 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.84943.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.142418.log.gz 2024-08-31T22:55:55.456 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136856.log 2024-08-31T22:55:55.457 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.160951.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107248.log 2024-08-31T22:55:55.457 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.160951.log.gz 2024-08-31T22:55:55.457 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.136856.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.163575.log.gz 2024-08-31T22:55:55.458 INFO:teuthology.orchestra.run.smithi032.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136856.log.gz 2024-08-31T22:55:55.458 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64932.log 2024-08-31T22:55:55.459 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110833.log 2024-08-31T22:55:55.459 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64932.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71599.log 2024-08-31T22:55:55.460 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.110833.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31053.log.gz 2024-08-31T22:55:55.460 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31557.log.gz 2024-08-31T22:55:55.460 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50914.log.gz 2024-08-31T22:55:55.460 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101944.log 2024-08-31T22:55:55.461 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.71599.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.164515.log 2024-08-31T22:55:55.461 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101944.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99921.log 2024-08-31T22:55:55.462 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.164515.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160091.log 2024-08-31T22:55:55.462 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.99921.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.164515.log.gz 2024-08-31T22:55:55.463 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30716.log 2024-08-31T22:55:55.463 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.160091.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77715.log 2024-08-31T22:55:55.463 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.30716.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30774.log 2024-08-31T22:55:55.464 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77715.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116954.log 2024-08-31T22:55:55.465 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.30774.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.160091.log.gz 2024-08-31T22:55:55.465 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35107.log 2024-08-31T22:55:55.465 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.116954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.163425.log 2024-08-31T22:55:55.465 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.35107.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115390.log 2024-08-31T22:55:55.467 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51883.log 2024-08-31T22:55:55.467 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.163425.log: /var/log/ceph/cluster2-client.mirror.115390.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30809.log 2024-08-31T22:55:55.467 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9%/var/log/ceph/cluster2-client.mirror.51883.log: -- replaced with /var/log/ceph/cluster1-client.mirror.163425.log.gz 2024-08-31T22:55:55.467 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41581.log 2024-08-31T22:55:55.467 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.30809.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39122.log 2024-08-31T22:55:55.469 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.41581.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.166058.log 2024-08-31T22:55:55.469 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39122.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30867.log 2024-08-31T22:55:55.469 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.166058.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87555.log 2024-08-31T22:55:55.469 INFO:teuthology.orchestra.run.smithi032.stderr: 26.4% -- replaced with /var/log/ceph/cluster2-client.mirror.166058.log.gz 2024-08-31T22:55:55.469 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.30867.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55491.log 2024-08-31T22:55:55.469 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65246.log 2024-08-31T22:55:55.470 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.87555.log: /var/log/ceph/cluster2-client.mirror.55491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158016.log 2024-08-31T22:55:55.471 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65246.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35014.log 2024-08-31T22:55:55.471 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/cluster2-client.mirror.158016.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120525.log 2024-08-31T22:55:55.471 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.35014.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98673.log 2024-08-31T22:55:55.471 INFO:teuthology.orchestra.run.smithi032.stderr:gzip 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.158016.log.gz -5 2024-08-31T22:55:55.471 INFO:teuthology.orchestra.run.smithi032.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.31336.log 2024-08-31T22:55:55.472 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.120525.log: /var/log/ceph/cluster2-client.mirror.98673.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75279.log 2024-08-31T22:55:55.472 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.31336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146078.log 2024-08-31T22:55:55.472 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121825.log 2024-08-31T22:55:55.473 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75279.log: /var/log/ceph/cluster1-client.mirror.146078.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52596.log 2024-08-31T22:55:55.473 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.154098.log 2024-08-31T22:55:55.473 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.146078.log.gz 2024-08-31T22:55:55.473 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.52596.log: /var/log/ceph/cluster2-client.mirror.121825.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28216.log 2024-08-31T22:55:55.473 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.154098.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80982.log 2024-08-31T22:55:55.474 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28216.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.154098.log.gz 2024-08-31T22:55:55.475 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28216.log.gz 2024-08-31T22:55:55.475 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120646.log 2024-08-31T22:55:55.475 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.80982.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56925.log 2024-08-31T22:55:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.120646.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115462.log 2024-08-31T22:55:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.56925.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54030.log 2024-08-31T22:55:55.477 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.115462.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65959.log 2024-08-31T22:55:55.477 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.54030.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147831.log 2024-08-31T22:55:55.478 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65959.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50194.log 2024-08-31T22:55:55.478 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.147831.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48869.log 2024-08-31T22:55:55.478 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50194.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68482.log.gz 2024-08-31T22:55:55.478 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28217.log 2024-08-31T22:55:55.479 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48869.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71134.log 2024-08-31T22:55:55.479 INFO:teuthology.orchestra.run.smithi032.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.147831.log.gz 2024-08-31T22:55:55.480 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28217.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28217.log.gz 2024-08-31T22:55:55.480 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29198.log 2024-08-31T22:55:55.481 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53379.log 2024-08-31T22:55:55.481 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.71134.log: /var/log/ceph/cluster1-client.mirror.29198.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166449.log 2024-08-31T22:55:55.482 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165718.log 2024-08-31T22:55:55.482 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53379.log: /var/log/ceph/cluster2-client.admin.166449.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166449.log.gz 2024-08-31T22:55:55.482 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44197.log 2024-08-31T22:55:55.482 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127099.log 2024-08-31T22:55:55.491 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165718.log: /var/log/ceph/cluster2-client.mirror.44197.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72033.log 2024-08-31T22:55:55.496 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.127099.log: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75279.log.gz 2024-08-31T22:55:55.502 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52817.log.gz 2024-08-31T22:55:55.503 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28749.log 2024-08-31T22:55:55.503 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70199.log.gz 2024-08-31T22:55:55.504 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36471.log.gz 2024-08-31T22:55:55.504 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.72033.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77715.log.gz 2024-08-31T22:55:55.517 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.103099.log.gz 2024-08-31T22:55:55.519 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29540.log 2024-08-31T22:55:55.519 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.28749.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59262.log.gz 2024-08-31T22:55:55.519 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28749.log.gz 2024-08-31T22:55:55.521 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.113333.log.gz 2024-08-31T22:55:55.523 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.105880.log.gz 2024-08-31T22:55:55.524 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.107248.log.gz 2024-08-31T22:55:55.524 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108027.log.gz 2024-08-31T22:55:55.527 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.89578.log.gz 2024-08-31T22:55:55.528 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81672.log.gz 2024-08-31T22:55:55.528 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.114457.log.gz 2024-08-31T22:55:55.529 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.116954.log.gz 2024-08-31T22:55:55.530 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.60723.log.gz 2024-08-31T22:55:55.531 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61037.log.gz 2024-08-31T22:55:55.531 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61223.log.gz 2024-08-31T22:55:55.533 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60630.log.gz 2024-08-31T22:55:55.533 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61878.log.gz 2024-08-31T22:55:55.534 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84943.log.gz 2024-08-31T22:55:55.535 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50194.log.gz 2024-08-31T22:55:55.536 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.87555.log.gz 2024-08-31T22:55:55.536 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66556.log.gz 2024-08-31T22:55:55.537 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68761.log.gz 2024-08-31T22:55:55.538 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69323.log.gz 2024-08-31T22:55:55.538 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69544.log.gz 2024-08-31T22:55:55.539 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71134.log.gz 2024-08-31T22:55:55.540 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111035.log 2024-08-31T22:55:55.541 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.29540.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71599.log.gz 2024-08-31T22:55:55.541 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72033.log.gz 2024-08-31T22:55:55.541 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101944.log.gz 2024-08-31T22:55:55.541 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104853.log.gz 2024-08-31T22:55:55.542 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102971.log.gz 2024-08-31T22:55:55.542 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.110833.log.gz 2024-08-31T22:55:55.543 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30716.log.gz 2024-08-31T22:55:55.543 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31429.log.gz 2024-08-31T22:55:55.544 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30809.log.gz 2024-08-31T22:55:55.544 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30867.log.gz 2024-08-31T22:55:55.544 INFO:teuthology.orchestra.run.smithi032.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111060.log.gz 2024-08-31T22:55:55.544 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30774.log.gz 2024-08-31T22:55:55.545 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64932.log.gz 2024-08-31T22:55:55.547 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.91161.log.gz 2024-08-31T22:55:55.549 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39401.log.gz 2024-08-31T22:55:55.550 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52596.log.gz 2024-08-31T22:55:55.551 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55491.log.gz 2024-08-31T22:55:55.551 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.107188.log.gz 2024-08-31T22:55:55.552 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39494.log 2024-08-31T22:55:55.552 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.112306.log.gz 2024-08-31T22:55:55.554 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.111035.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.53809.log.gz 2024-08-31T22:55:55.556 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105694.log.gz 2024-08-31T22:55:55.562 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44980.log.gz 2024-08-31T22:55:55.562 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45038.log.gz 2024-08-31T22:55:55.563 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56925.log.gz 2024-08-31T22:55:55.565 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99921.log.gz 2024-08-31T22:55:55.567 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65246.log.gz 2024-08-31T22:55:55.568 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.53379.log.gz 2024-08-31T22:55:55.570 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.101231.log.gz 2024-08-31T22:55:55.573 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.54030.log.gz 2024-08-31T22:55:55.573 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39122.log.gz 2024-08-31T22:55:55.574 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44197.log.gz 2024-08-31T22:55:55.574 INFO:teuthology.orchestra.run.smithi032.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.120525.log.gz 2024-08-31T22:55:55.574 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111035.log.gz 2024-08-31T22:55:55.574 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105787.log.gz 2024-08-31T22:55:55.574 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31336.log.gz 2024-08-31T22:55:55.578 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81730.log 2024-08-31T22:55:55.579 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39494.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.115462.log.gz 2024-08-31T22:55:55.580 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47559.log.gz 2024-08-31T22:55:55.582 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.84346.log.gz 2024-08-31T22:55:55.582 INFO:teuthology.orchestra.run.smithi032.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.124574.log.gz 2024-08-31T22:55:55.583 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113988.log 2024-08-31T22:55:55.584 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.81730.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40583.log 2024-08-31T22:55:55.584 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99421.log.gz 2024-08-31T22:55:55.584 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48869.log.gz 2024-08-31T22:55:55.585 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.81730.log.gz 2024-08-31T22:55:55.585 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.98673.log.gz 2024-08-31T22:55:55.585 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.113988.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42271.log 2024-08-31T22:55:55.586 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.40583.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80180.log 2024-08-31T22:55:55.586 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.42271.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48249.log 2024-08-31T22:55:55.587 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.80180.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82199.log 2024-08-31T22:55:55.588 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48249.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100820.log 2024-08-31T22:55:55.589 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82199.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95906.log 2024-08-31T22:55:55.589 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.100820.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130051.log 2024-08-31T22:55:55.590 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.95906.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37405.log 2024-08-31T22:55:55.591 INFO:teuthology.orchestra.run.smithi032.stderr: 56.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29198.log.gz 2024-08-31T22:55:55.591 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.130051.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130051.log.gz 2024-08-31T22:55:55.591 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65959.log.gz 2024-08-31T22:55:55.595 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118992.log 2024-08-31T22:55:55.595 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.37405.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29062.log 2024-08-31T22:55:55.596 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118992.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101979.log 2024-08-31T22:55:55.597 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.29062.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158420.log 2024-08-31T22:55:55.598 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101979.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163845.log 2024-08-31T22:55:55.599 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158420.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.158420.log.gz 2024-08-31T22:55:55.603 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101979.log.gz 2024-08-31T22:55:55.603 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37405.log.gz 2024-08-31T22:55:55.608 INFO:teuthology.orchestra.run.smithi032.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.admin.29301.log.gz 2024-08-31T22:55:55.608 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41581.log.gz 2024-08-31T22:55:55.608 INFO:teuthology.orchestra.run.smithi032.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.121245.log.gz 2024-08-31T22:55:55.608 INFO:teuthology.orchestra.run.smithi032.stderr: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.29062.log.gz 2024-08-31T22:55:55.608 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115927.log 2024-08-31T22:55:55.608 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163845.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39494.log.gz 2024-08-31T22:55:55.609 INFO:teuthology.orchestra.run.smithi032.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.62750.log.gz 2024-08-31T22:55:55.609 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.163845.log.gz 2024-08-31T22:55:55.610 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94003.log 2024-08-31T22:55:55.610 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.115927.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80703.log 2024-08-31T22:55:55.610 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94003.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.149023.log.gz 2024-08-31T22:55:55.610 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58700.log 2024-08-31T22:55:55.610 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119474.log 2024-08-31T22:55:55.611 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.80703.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106500.log 2024-08-31T22:55:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58700.log: /var/log/ceph/cluster2-client.mirror.119474.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117774.log.gz 2024-08-31T22:55:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34731.log 2024-08-31T22:55:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.106500.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72816.log 2024-08-31T22:55:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.34731.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118701.log 2024-08-31T22:55:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.72816.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30902.log 2024-08-31T22:55:55.618 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.118701.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88830.log.gz 2024-08-31T22:55:55.619 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151544.log 2024-08-31T22:55:55.621 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.30902.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51697.log 2024-08-31T22:55:55.621 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.151544.log: gzip -5 88.0% --verbose -- -- replaced with /var/log/ceph/cluster2-client.mirror.40583.log.gz /var/log/ceph/cluster2-client.mirror.84815.log 2024-08-31T22:55:55.621 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:55.621 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63475.log 2024-08-31T22:55:55.621 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151544.log.gz 2024-08-31T22:55:55.621 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51697.log: /var/log/ceph/cluster2-client.mirror.84815.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61785.log 2024-08-31T22:55:55.621 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63475.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49241.log 2024-08-31T22:55:55.622 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62875.log 2024-08-31T22:55:55.622 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49241.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104946.log 2024-08-31T22:55:55.622 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.62875.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118908.log 2024-08-31T22:55:55.622 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.104946.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56053.log 2024-08-31T22:55:55.623 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118908.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93724.log 2024-08-31T22:55:55.624 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.56053.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37219.log 2024-08-31T22:55:55.624 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.93724.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94131.log 2024-08-31T22:55:55.624 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.37219.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89202.log 2024-08-31T22:55:55.624 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49745.log 2024-08-31T22:55:55.626 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89202.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105283.log 2024-08-31T22:55:55.626 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49745.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120253.log 2024-08-31T22:55:55.626 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70536.log 2024-08-31T22:55:55.626 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.120253.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61316.log 2024-08-31T22:55:55.627 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.70536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.164237.log 2024-08-31T22:55:55.627 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61316.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151611.log 2024-08-31T22:55:55.627 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.164237.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75340.log 2024-08-31T22:55:55.628 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.151611.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113054.log 2024-08-31T22:55:55.628 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.164237.log.gz 2024-08-31T22:55:55.630 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.75340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89016.log 2024-08-31T22:55:55.631 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112682.log 2024-08-31T22:55:55.632 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.113054.log: /var/log/ceph/cluster2-client.mirror.89016.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.80982.log.gz 2024-08-31T22:55:55.632 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.112682.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80552.log 2024-08-31T22:55:55.632 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53995.log 2024-08-31T22:55:55.633 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.80552.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75257.log 2024-08-31T22:55:55.633 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-08-31T22:55:55.634 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53995.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90898.log 2024-08-31T22:55:55.634 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.75257.log: /var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29258.log 2024-08-31T22:55:55.634 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.90898.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34510.log 2024-08-31T22:55:55.635 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.29258.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29258.log.gz 2024-08-31T22:55:55.636 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.117516.log.gz 2024-08-31T22:55:55.636 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.34510.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69788.log 2024-08-31T22:55:55.636 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158662.log 2024-08-31T22:55:55.637 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.69788.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.93724.log.gz 2024-08-31T22:55:55.637 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100390.log 2024-08-31T22:55:55.637 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.113988.log.gz 2024-08-31T22:55:55.638 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158662.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.158662.log.gz 2024-08-31T22:55:55.639 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137879.log 2024-08-31T22:55:55.639 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.100390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115362.log 2024-08-31T22:55:55.641 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112492.log 2024-08-31T22:55:55.641 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.137879.log: /var/log/ceph/cluster1-client.admin.115362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147042.log 2024-08-31T22:55:55.641 INFO:teuthology.orchestra.run.smithi032.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.117732.log.gz 2024-08-31T22:55:55.641 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.112492.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33239.log 2024-08-31T22:55:55.643 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.147042.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.137879.log.gz 2024-08-31T22:55:55.643 INFO:teuthology.orchestra.run.smithi032.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118818.log.gz 2024-08-31T22:55:55.644 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.147042.log.gz 2024-08-31T22:55:55.644 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42887.log 2024-08-31T22:55:55.644 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33239.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159907.log 2024-08-31T22:55:55.646 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.42887.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152276.log 2024-08-31T22:55:55.646 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159907.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.159907.log.gz 2024-08-31T22:55:55.646 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43728.log 2024-08-31T22:55:55.646 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.152276.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.152276.log.gz 2024-08-31T22:55:55.647 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150436.log 2024-08-31T22:55:55.648 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.43728.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103765.log 2024-08-31T22:55:55.649 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.150436.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.150436.log.gz 2024-08-31T22:55:55.650 INFO:teuthology.orchestra.run.smithi032.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.111107.log.gz 2024-08-31T22:55:55.651 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67269.log 2024-08-31T22:55:55.651 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.80703.log.gz 2024-08-31T22:55:55.653 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.103765.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74742.log 2024-08-31T22:55:55.653 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118028.log 2024-08-31T22:55:55.653 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137222.log 2024-08-31T22:55:55.654 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.74742.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43205.log.gz 2024-08-31T22:55:55.655 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.118028.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83540.log 2024-08-31T22:55:55.655 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.137222.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63475.log.gz 2024-08-31T22:55:55.655 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.82199.log.gz 2024-08-31T22:55:55.657 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61316.log.gz 2024-08-31T22:55:55.657 INFO:teuthology.orchestra.run.smithi032.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.137222.log.gz 2024-08-31T22:55:55.657 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158278.log 2024-08-31T22:55:55.657 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83540.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159696.log 2024-08-31T22:55:55.658 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61785.log.gz 2024-08-31T22:55:55.663 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158278.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.158278.log.gz 2024-08-31T22:55:55.663 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76537.log 2024-08-31T22:55:55.663 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159696.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.106500.log.gz 2024-08-31T22:55:55.663 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104946.log.gz 2024-08-31T22:55:55.663 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.112682.log.gz 2024-08-31T22:55:55.663 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30902.log.gz 2024-08-31T22:55:55.663 INFO:teuthology.orchestra.run.smithi032.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119474.log.gz 2024-08-31T22:55:55.664 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100390.log.gz 2024-08-31T22:55:55.664 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94003.log.gz 2024-08-31T22:55:55.664 INFO:teuthology.orchestra.run.smithi032.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.159696.log.gz 2024-08-31T22:55:55.664 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94131.log.gz 2024-08-31T22:55:55.664 INFO:teuthology.orchestra.run.smithi032.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118278.log.gz 2024-08-31T22:55:55.664 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29318.log 2024-08-31T22:55:55.665 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.76537.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33948.log 2024-08-31T22:55:55.665 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.29318.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84253.log 2024-08-31T22:55:55.665 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50763.log 2024-08-31T22:55:55.668 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33948.log: /var/log/ceph/cluster2-client.mirror.84253.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.75340.log.gz 2024-08-31T22:55:55.669 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50763.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142929.log 2024-08-31T22:55:55.669 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.113054.log.gz 2024-08-31T22:55:55.669 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112085.log 2024-08-31T22:55:55.669 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.142929.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162368.log 2024-08-31T22:55:55.669 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.142929.log.gz/var/log/ceph/cluster2-client.mirror.112085.log: 2024-08-31T22:55:55.669 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159625.log 2024-08-31T22:55:55.669 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56367.log 2024-08-31T22:55:55.669 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162368.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.162368.log.gz 2024-08-31T22:55:55.669 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159625.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84439.log 2024-08-31T22:55:55.669 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.56367.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47373.log 2024-08-31T22:55:55.669 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.84439.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79862.log 2024-08-31T22:55:55.669 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.47373.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152979.log 2024-08-31T22:55:55.670 INFO:teuthology.orchestra.run.smithi032.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.159625.log.gz/var/log/ceph/cluster2-client.mirror.79862.log: 2024-08-31T22:55:55.670 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64781.log 2024-08-31T22:55:55.672 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.152979.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.152979.log.gz 2024-08-31T22:55:55.672 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117330.log 2024-08-31T22:55:55.672 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64781.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160748.log 2024-08-31T22:55:55.672 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.117330.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34417.log.gz 2024-08-31T22:55:55.672 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78215.log 2024-08-31T22:55:55.672 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0%/var/log/ceph/cluster1-client.mirror.160748.log: -- replaced with /var/log/ceph/cluster2-client.mirror.34638.log.gz 2024-08-31T22:55:55.673 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34731.log.gz 2024-08-31T22:55:55.673 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34510.log.gz 2024-08-31T22:55:55.673 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.160748.log.gz 2024-08-31T22:55:55.673 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.35014.log.gz 2024-08-31T22:55:55.673 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35107.log.gz 2024-08-31T22:55:55.673 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106814.log 2024-08-31T22:55:55.674 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.78215.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157005.log 2024-08-31T22:55:55.674 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.106814.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.115362.log.gz 2024-08-31T22:55:55.674 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42271.log.gz 2024-08-31T22:55:55.674 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120780.log 2024-08-31T22:55:55.674 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.89016.log.gz 2024-08-31T22:55:55.675 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157005.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.157005.log.gz 2024-08-31T22:55:55.675 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70536.log.gz 2024-08-31T22:55:55.675 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.89202.log.gz 2024-08-31T22:55:55.676 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102130.log 2024-08-31T22:55:55.676 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.120780.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32863.log 2024-08-31T22:55:55.676 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.102130.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.166083.log 2024-08-31T22:55:55.677 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.32863.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158371.log 2024-08-31T22:55:55.680 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.166083.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.166083.log.gz 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121619.log.gz 2024-08-31T22:55:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142954.log 2024-08-31T22:55:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58607.log 2024-08-31T22:55:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158371.log: /var/log/ceph/cluster2-client.mirror.142954.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.112492.log.gz 2024-08-31T22:55:55.681 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.158371.log.gz 2024-08-31T22:55:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153789.log 2024-08-31T22:55:55.682 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58607.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.112085.log.gz 2024-08-31T22:55:55.682 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:55.682 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90538.log 2024-08-31T22:55:55.682 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98394.log 2024-08-31T22:55:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.153789.log: /var/log/ceph/cluster2-client.mirror.90538.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.142954.log.gz 2024-08-31T22:55:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41616.log 2024-08-31T22:55:55.684 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.98394.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.78215.log.gz 2024-08-31T22:55:55.684 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76537.log.gz 2024-08-31T22:55:55.684 INFO:teuthology.orchestra.run.smithi032.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.153789.log.gz 2024-08-31T22:55:55.684 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33948.log.gz 2024-08-31T22:55:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49838.log 2024-08-31T22:55:55.686 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48776.log 2024-08-31T22:55:55.686 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.41616.log: /var/log/ceph/cluster2-client.mirror.49838.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151947.log 2024-08-31T22:55:55.686 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48776.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68610.log 2024-08-31T22:55:55.686 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86586.log 2024-08-31T22:55:55.687 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.151947.log: /var/log/ceph/cluster2-client.mirror.68610.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101382.log 2024-08-31T22:55:55.687 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86586.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158320.log 2024-08-31T22:55:55.687 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101382.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90695.log 2024-08-31T22:55:55.688 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.80552.log.gz 2024-08-31T22:55:55.688 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158320.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.151947.log.gz 2024-08-31T22:55:55.688 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118748.log 2024-08-31T22:55:55.688 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69788.log.gz 2024-08-31T22:55:55.689 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.83540.log.gz 2024-08-31T22:55:55.689 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.90695.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98394.log.gz 2024-08-31T22:55:55.690 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143315.log 2024-08-31T22:55:55.690 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109337.log 2024-08-31T22:55:55.691 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.118748.log: /var/log/ceph/cluster1-client.mirror.143315.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69823.log 2024-08-31T22:55:55.691 INFO:teuthology.orchestra.run.smithi032.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.158320.log.gz 2024-08-31T22:55:55.691 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143315.log.gz 2024-08-31T22:55:55.692 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94751.log 2024-08-31T22:55:55.692 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.109337.log: /var/log/ceph/cluster2-client.mirror.69823.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136878.log 2024-08-31T22:55:55.692 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94751.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117593.log 2024-08-31T22:55:55.692 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147806.log 2024-08-31T22:55:55.692 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.136878.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59227.log 2024-08-31T22:55:55.694 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.147806.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.62875.log.gz 2024-08-31T22:55:55.694 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.117593.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.147806.log.gz 2024-08-31T22:55:55.694 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89520.log 2024-08-31T22:55:55.694 INFO:teuthology.orchestra.run.smithi032.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.136878.log.gz/var/log/ceph/cluster2-client.mirror.59227.log: 2024-08-31T22:55:55.694 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142974.log 2024-08-31T22:55:55.694 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89520.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153744.log 2024-08-31T22:55:55.694 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.142974.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115259.log 2024-08-31T22:55:55.694 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.153744.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103471.log 2024-08-31T22:55:55.696 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0%/var/log/ceph/cluster2-client.mirror.115259.log: -- replaced with /var/log/ceph/cluster1-client.mirror.153744.log.gz 2024-08-31T22:55:55.696 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49555.log.gz 2024-08-31T22:55:55.696 INFO:teuthology.orchestra.run.smithi032.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.142974.log.gz 2024-08-31T22:55:55.696 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55991.log 2024-08-31T22:55:55.696 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.103471.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69509.log 2024-08-31T22:55:55.700 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.55991.log: --verbose -- /var/log/ceph/cluster2-client.mirror.112740.log 2024-08-31T22:55:55.700 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.51883.log.gz 2024-08-31T22:55:55.700 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33669.log 2024-08-31T22:55:55.700 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.69509.log: /var/log/ceph/cluster2-client.mirror.112740.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115282.log 2024-08-31T22:55:55.700 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33669.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96243.log 2024-08-31T22:55:55.700 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157967.log 2024-08-31T22:55:55.700 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.115282.log: /var/log/ceph/cluster2-client.mirror.96243.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81172.log 2024-08-31T22:55:55.700 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157967.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117822.log 2024-08-31T22:55:55.700 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.157967.log.gz 2024-08-31T22:55:55.700 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51511.log 2024-08-31T22:55:55.701 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.81172.log: /var/log/ceph/cluster2-client.mirror.117822.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82106.log 2024-08-31T22:55:55.701 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118658.log 2024-08-31T22:55:55.701 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33390.log 2024-08-31T22:55:55.702 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82106.log: /var/log/ceph/cluster1-client.mirror.118658.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32363.log 2024-08-31T22:55:55.702 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63754.log 2024-08-31T22:55:55.702 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58700.log.gz 2024-08-31T22:55:55.702 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.32363.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53995.log.gz 2024-08-31T22:55:55.702 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.79862.log.gz 2024-08-31T22:55:55.702 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69509.log.gz 2024-08-31T22:55:55.702 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161333.log 2024-08-31T22:55:55.702 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63754.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152712.log 2024-08-31T22:55:55.703 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64781.log.gz 2024-08-31T22:55:55.703 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161333.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.161333.log.gz 2024-08-31T22:55:55.704 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162706.log 2024-08-31T22:55:55.704 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.152712.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-08-31T22:55:55.704 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162706.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.152712.log.gz 2024-08-31T22:55:55.704 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.162706.log.gz 2024-08-31T22:55:55.705 INFO:teuthology.orchestra.run.smithi032.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.130481.log.gz 2024-08-31T22:55:55.705 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84067.log 2024-08-31T22:55:55.706 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66463.log 2024-08-31T22:55:55.707 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.84067.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61095.log 2024-08-31T22:55:55.707 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158158.log 2024-08-31T22:55:55.707 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.66463.log: /var/log/ceph/cluster2-client.mirror.61095.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70855.log 2024-08-31T22:55:55.707 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.102130.log.gz 2024-08-31T22:55:55.708 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158158.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97960.log 2024-08-31T22:55:55.708 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.70855.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.158158.log.gz 2024-08-31T22:55:55.708 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92728.log 2024-08-31T22:55:55.709 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102502.log 2024-08-31T22:55:55.709 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.97960.log: /var/log/ceph/cluster2-client.mirror.92728.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47501.log 2024-08-31T22:55:55.709 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.102502.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137713.log 2024-08-31T22:55:55.709 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.47501.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.151611.log.gz 2024-08-31T22:55:55.709 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112775.log 2024-08-31T22:55:55.710 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.137713.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110329.log 2024-08-31T22:55:55.710 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.112775.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33239.log.gz 2024-08-31T22:55:55.711 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.106814.log.gz 2024-08-31T22:55:55.711 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129252.log 2024-08-31T22:55:55.712 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.110329.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151922.log 2024-08-31T22:55:55.712 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.129252.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159097.log 2024-08-31T22:55:55.712 INFO:teuthology.orchestra.run.smithi032.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.137713.log.gz 2024-08-31T22:55:55.712 INFO:teuthology.orchestra.run.smithi032.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129252.log.gz 2024-08-31T22:55:55.712 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.151922.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83226.log 2024-08-31T22:55:55.712 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159097.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90171.log 2024-08-31T22:55:55.712 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151922.log.gz 2024-08-31T22:55:55.713 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83226.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84195.log 2024-08-31T22:55:55.715 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58049.log 2024-08-31T22:55:55.715 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.90171.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.159097.log.gz 2024-08-31T22:55:55.715 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.84195.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61130.log 2024-08-31T22:55:55.715 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58049.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34359.log 2024-08-31T22:55:55.715 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61130.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110891.log 2024-08-31T22:55:55.715 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.34359.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35665.log 2024-08-31T22:55:55.715 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160771.log 2024-08-31T22:55:55.715 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.110891.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52631.log 2024-08-31T22:55:55.715 INFO:teuthology.orchestra.run.smithi032.stderr: 71.0% -- replaced with /var/log/ceph/cluster2-client.admin.29318.log.gz 2024-08-31T22:55:55.716 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.160771.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96991.log 2024-08-31T22:55:55.716 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.160771.log.gz 2024-08-31T22:55:55.716 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50763.log.gz 2024-08-31T22:55:55.716 INFO:teuthology.orchestra.run.smithi032.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118638.log.gz 2024-08-31T22:55:55.716 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.52631.log: /var/log/ceph/cluster2-client.mirror.35665.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39928.log 2024-08-31T22:55:55.716 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96991.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158846.log 2024-08-31T22:55:55.721 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135059.log 2024-08-31T22:55:55.722 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39928.log: /var/log/ceph/cluster2-client.mirror.158846.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58665.log 2024-08-31T22:55:55.722 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.135059.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145247.log 2024-08-31T22:55:55.722 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65680.log 2024-08-31T22:55:55.722 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.135059.log.gz 2024-08-31T22:55:55.722 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58665.log: /var/log/ceph/cluster2-client.mirror.145247.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90403.log 2024-08-31T22:55:55.722 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65680.log: gzip 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.145247.log.gz -5 2024-08-31T22:55:55.722 INFO:teuthology.orchestra.run.smithi032.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.164085.log 2024-08-31T22:55:55.722 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.90403.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84815.log.gz 2024-08-31T22:55:55.722 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.84067.log.gz 2024-08-31T22:55:55.722 INFO:teuthology.orchestra.run.smithi032.stderr: 90.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109806.log 2024-08-31T22:55:55.723 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.158846.log.gz 2024-08-31T22:55:55.723 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164085.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97646.log 2024-08-31T22:55:55.723 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.109806.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80924.log 2024-08-31T22:55:55.723 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.97646.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.84195.log.gz 2024-08-31T22:55:55.723 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84253.log.gz 2024-08-31T22:55:55.723 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.80924.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84439.log.gz 2024-08-31T22:55:55.723 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54685.log 2024-08-31T22:55:55.723 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101568.log 2024-08-31T22:55:55.723 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.112775.log.gz 2024-08-31T22:55:55.723 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.54685.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90770.log 2024-08-31T22:55:55.723 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59824.log 2024-08-31T22:55:55.723 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101568.log: /var/log/ceph/cluster2-client.mirror.90770.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144518.log 2024-08-31T22:55:55.723 INFO:teuthology.orchestra.run.smithi032.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.164085.log.gz 2024-08-31T22:55:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.59824.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50298.log 2024-08-31T22:55:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.144518.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125045.log 2024-08-31T22:55:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50298.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101510.log 2024-08-31T22:55:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.125045.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69079.log 2024-08-31T22:55:55.727 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101510.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.144518.log.gz 2024-08-31T22:55:55.727 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105097.log 2024-08-31T22:55:55.727 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.69079.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118208.log 2024-08-31T22:55:55.727 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105097.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86776.log 2024-08-31T22:55:55.727 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.118208.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95065.log 2024-08-31T22:55:55.727 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86776.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90043.log 2024-08-31T22:55:55.727 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.29552.log 2024-08-31T22:55:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.95065.log: /var/log/ceph/cluster2-client.mirror.90043.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84532.log 2024-08-31T22:55:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140114.log 2024-08-31T22:55:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.0.29552.log: /var/log/ceph/cluster2-client.mirror.84532.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47652.log 2024-08-31T22:55:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.140114.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60254.log 2024-08-31T22:55:55.728 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.140114.log.gz 2024-08-31T22:55:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.47652.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33855.log 2024-08-31T22:55:55.728 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51697.log.gz 2024-08-31T22:55:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60254.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93569.log 2024-08-31T22:55:55.728 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.56053.log.gz 2024-08-31T22:55:55.728 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55991.log.gz 2024-08-31T22:55:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33855.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78370.log 2024-08-31T22:55:55.731 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156017.log 2024-08-31T22:55:55.731 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.93569.log: /var/log/ceph/cluster2-client.mirror.78370.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64312.log 2024-08-31T22:55:55.731 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156017.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98859.log 2024-08-31T22:55:55.731 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.156017.log.gz 2024-08-31T22:55:55.731 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64312.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153046.log 2024-08-31T22:55:55.731 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.98859.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112271.log 2024-08-31T22:55:55.731 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56367.log.gz 2024-08-31T22:55:55.731 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.153046.log: 88.1% 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58607.log.gz 2024-08-31T22:55:55.731 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.58665.log.gz 2024-08-31T22:55:55.731 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68610.log.gz 2024-08-31T22:55:55.731 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41616.log.gz 2024-08-31T22:55:55.731 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52631.log.gz 2024-08-31T22:55:55.731 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32363.log.gz 2024-08-31T22:55:55.732 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32863.log.gz 2024-08-31T22:55:55.732 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33390.log.gz 2024-08-31T22:55:55.732 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35665.log.gz 2024-08-31T22:55:55.733 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.109337.log.gz 2024-08-31T22:55:55.733 INFO:teuthology.orchestra.run.smithi032.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.118701.log.gz 2024-08-31T22:55:55.733 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.110329.log.gz 2024-08-31T22:55:55.734 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48249.log.gz 2024-08-31T22:55:55.734 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81172.log.gz 2024-08-31T22:55:55.735 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37033.log 2024-08-31T22:55:55.735 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.90403.log.gz 2024-08-31T22:55:55.737 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134728.log 2024-08-31T22:55:55.737 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33855.log.gz 2024-08-31T22:55:55.737 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.112271.log: /var/log/ceph/cluster2-client.mirror.37033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161086.log 2024-08-31T22:55:55.737 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.134728.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.92728.log.gz 2024-08-31T22:55:55.737 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131116.log 2024-08-31T22:55:55.737 INFO:teuthology.orchestra.run.smithi032.stderr: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.134728.log.gz 2024-08-31T22:55:55.737 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161086.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125467.log 2024-08-31T22:55:55.737 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.161086.log.gz 2024-08-31T22:55:55.737 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.131116.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131116.log.gz 2024-08-31T22:55:55.739 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162121.log 2024-08-31T22:55:55.739 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.125467.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.27909.log 2024-08-31T22:55:55.741 INFO:teuthology.orchestra.run.smithi032.stderr: 84.7%/var/log/ceph/cluster1-client.mirror.162121.log: -- replaced with /var/log/ceph/cluster2-client.mirror.125467.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52220.log 2024-08-31T22:55:55.741 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:55.741 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.162121.log.gz 2024-08-31T22:55:55.741 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.27909.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57580.log 2024-08-31T22:55:55.742 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.27909.log.gz 2024-08-31T22:55:55.742 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.52220.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105004.log 2024-08-31T22:55:55.742 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.57580.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86342.log 2024-08-31T22:55:55.742 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105004.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122138.log 2024-08-31T22:55:55.742 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86342.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43391.log 2024-08-31T22:55:55.742 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.122138.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110178.log 2024-08-31T22:55:55.746 INFO:teuthology.orchestra.run.smithi032.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.122138.log.gz/var/log/ceph/cluster2-client.mirror.43391.log: 2024-08-31T22:55:55.746 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111992.log 2024-08-31T22:55:55.746 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118298.log 2024-08-31T22:55:55.746 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.110178.log: /var/log/ceph/cluster2-client.mirror.111992.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75599.log 2024-08-31T22:55:55.746 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.118298.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79711.log 2024-08-31T22:55:55.747 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75599.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70630.log 2024-08-31T22:55:55.747 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.79711.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116710.log 2024-08-31T22:55:55.747 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.70630.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118458.log 2024-08-31T22:55:55.747 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.116710.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156834.log 2024-08-31T22:55:55.747 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118458.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47935.log 2024-08-31T22:55:55.747 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156834.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155503.log 2024-08-31T22:55:55.747 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.47935.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.95065.log.gz 2024-08-31T22:55:55.747 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.96991.log.gz 2024-08-31T22:55:55.750 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28624.log 2024-08-31T22:55:55.750 INFO:teuthology.orchestra.run.smithi032.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.156834.log.gz 2024-08-31T22:55:55.750 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.97646.log.gz 2024-08-31T22:55:55.750 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.155503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151967.log 2024-08-31T22:55:55.751 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28624.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28624.log.gz 2024-08-31T22:55:55.751 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43449.log 2024-08-31T22:55:55.751 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.151967.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.155503.log.gz 2024-08-31T22:55:55.751 INFO:teuthology.orchestra.run.smithi032.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.151967.log.gz 2024-08-31T22:55:55.753 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162773.log 2024-08-31T22:55:55.753 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.43449.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157754.log 2024-08-31T22:55:55.753 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162773.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61936.log 2024-08-31T22:55:55.753 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.162773.log.gz 2024-08-31T22:55:55.753 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157754.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.117330.log.gz 2024-08-31T22:55:55.753 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.157754.log.gz 2024-08-31T22:55:55.754 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101789.log 2024-08-31T22:55:55.754 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61936.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117593.log.gz 2024-08-31T22:55:55.755 INFO:teuthology.orchestra.run.smithi032.stderr:gzip 88.6% -5 -- replaced with /var/log/ceph/cluster2-client.mirror.98859.log.gz --verbose 2024-08-31T22:55:55.755 INFO:teuthology.orchestra.run.smithi032.stderr: -- /var/log/ceph/cluster2-client.mirror.78835.log 2024-08-31T22:55:55.755 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101789.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76723.log 2024-08-31T22:55:55.757 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.78835.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114174.log 2024-08-31T22:55:55.757 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57580.log.gz 2024-08-31T22:55:55.757 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54685.log.gz 2024-08-31T22:55:55.757 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65680.log.gz 2024-08-31T22:55:55.758 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.76723.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.61130.log.gz 2024-08-31T22:55:55.758 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61936.log.gz 2024-08-31T22:55:55.758 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135116.log 2024-08-31T22:55:55.758 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70855.log.gz 2024-08-31T22:55:55.758 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119416.log 2024-08-31T22:55:55.758 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.114174.log: /var/log/ceph/cluster2-client.mirror.135116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52689.log 2024-08-31T22:55:55.760 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.119416.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135116.log.gz 2024-08-31T22:55:55.760 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117144.log 2024-08-31T22:55:55.760 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.52689.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160353.log 2024-08-31T22:55:55.760 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.117144.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97367.log 2024-08-31T22:55:55.760 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.160353.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89892.log 2024-08-31T22:55:55.760 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.97367.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.160353.log.gz 2024-08-31T22:55:55.760 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103471.log.gz 2024-08-31T22:55:55.760 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162346.log 2024-08-31T22:55:55.760 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89892.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79114.log 2024-08-31T22:55:55.762 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162346.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157916.log 2024-08-31T22:55:55.762 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.79114.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101789.log.gz 2024-08-31T22:55:55.762 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.162346.log.gz 2024-08-31T22:55:55.762 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156350.log 2024-08-31T22:55:55.762 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66463.log.gz 2024-08-31T22:55:55.762 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157916.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39928.log.gz 2024-08-31T22:55:55.763 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42236.log 2024-08-31T22:55:55.764 INFO:teuthology.orchestra.run.smithi032.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.157916.log.gz/var/log/ceph/cluster2-client.mirror.156350.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42980.log 2024-08-31T22:55:55.764 INFO:teuthology.orchestra.run.smithi032.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.120780.log.gz 2024-08-31T22:55:55.764 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:55.764 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.42236.log: -- replaced with /var/log/ceph/cluster2-client.mirror.156350.log.gz 2024-08-31T22:55:55.764 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136433.log 2024-08-31T22:55:55.765 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.42980.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43263.log 2024-08-31T22:55:55.765 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.136433.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61692.log 2024-08-31T22:55:55.765 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.110891.log.gz 2024-08-31T22:55:55.765 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.43263.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136433.log.gz 2024-08-31T22:55:55.765 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42887.log.gz 2024-08-31T22:55:55.766 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.119416.log.gz 2024-08-31T22:55:55.766 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163280.log 2024-08-31T22:55:55.766 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61692.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117628.log 2024-08-31T22:55:55.767 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163280.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118055.log 2024-08-31T22:55:55.767 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.117628.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66986.log 2024-08-31T22:55:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118055.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88679.log 2024-08-31T22:55:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.66986.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29403.log 2024-08-31T22:55:55.769 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.88679.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25219.log 2024-08-31T22:55:55.769 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.112740.log.gz 2024-08-31T22:55:55.771 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.29403.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67269.log.gz 2024-08-31T22:55:55.771 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.89520.log.gz 2024-08-31T22:55:55.773 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.80180.log.gz 2024-08-31T22:55:55.773 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87090.log 2024-08-31T22:55:55.774 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30305.log 2024-08-31T22:55:55.774 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.25219.log: /var/log/ceph/cluster2-client.mirror.87090.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25219.log.gz 2024-08-31T22:55:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90322.log 2024-08-31T22:55:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.30305.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.105097.log.gz 2024-08-31T22:55:55.775 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.105004.log.gz 2024-08-31T22:55:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111465.log 2024-08-31T22:55:55.776 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82106.log.gz 2024-08-31T22:55:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.90322.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115683.log 2024-08-31T22:55:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67021.log 2024-08-31T22:55:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.111465.log: /var/log/ceph/cluster2-client.mirror.115683.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123042.log 2024-08-31T22:55:55.777 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59227.log.gz 2024-08-31T22:55:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67021.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118368.log 2024-08-31T22:55:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123019.log 2024-08-31T22:55:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.123042.log: /var/log/ceph/cluster2-client.mirror.118368.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60254.log.gz 2024-08-31T22:55:55.781 INFO:teuthology.orchestra.run.smithi032.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118908.log.gz 2024-08-31T22:55:55.782 INFO:teuthology.orchestra.run.smithi032.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.120253.log.gz 2024-08-31T22:55:55.782 INFO:teuthology.orchestra.run.smithi032.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118992.log.gz 2024-08-31T22:55:55.783 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69823.log.gz 2024-08-31T22:55:55.784 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72816.log.gz 2024-08-31T22:55:55.785 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37033.log.gz 2024-08-31T22:55:55.785 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.112271.log.gz 2024-08-31T22:55:55.785 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.78835.log.gz 2024-08-31T22:55:55.789 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.90322.log.gz 2024-08-31T22:55:55.790 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.114174.log.gz 2024-08-31T22:55:55.790 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30305.log.gz 2024-08-31T22:55:55.791 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55305.log 2024-08-31T22:55:55.791 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.123019.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123019.log.gz 2024-08-31T22:55:55.792 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146345.log 2024-08-31T22:55:55.793 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.55305.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34359.log.gz 2024-08-31T22:55:55.793 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52034.log 2024-08-31T22:55:55.794 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.146345.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100820.log.gz 2024-08-31T22:55:55.795 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101568.log.gz 2024-08-31T22:55:55.796 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42236.log.gz 2024-08-31T22:55:55.797 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76723.log.gz 2024-08-31T22:55:55.798 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105283.log.gz 2024-08-31T22:55:55.798 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.117628.log.gz 2024-08-31T22:55:55.798 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87090.log.gz 2024-08-31T22:55:55.798 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.86586.log.gz 2024-08-31T22:55:55.801 INFO:teuthology.orchestra.run.smithi032.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.121825.log.gz 2024-08-31T22:55:55.801 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137242.log 2024-08-31T22:55:55.802 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.52034.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117179.log 2024-08-31T22:55:55.802 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.95906.log.gz 2024-08-31T22:55:55.803 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.137242.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.96243.log.gz 2024-08-31T22:55:55.803 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.88679.log.gz 2024-08-31T22:55:55.805 INFO:teuthology.orchestra.run.smithi032.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.137242.log.gz 2024-08-31T22:55:55.805 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.115927.log.gz 2024-08-31T22:55:55.805 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose 88.5% -- -- replaced with /var/log/ceph/cluster2-client.mirror.61095.log.gz /var/log/ceph/cluster1-client.mirror.148956.log 2024-08-31T22:55:55.805 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:55.806 INFO:teuthology.orchestra.run.smithi032.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.127099.log.gz 2024-08-31T22:55:55.806 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61692.log.gz 2024-08-31T22:55:55.806 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.117179.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64312.log.gz 2024-08-31T22:55:55.806 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126609.log 2024-08-31T22:55:55.806 INFO:teuthology.orchestra.run.smithi032.stderr: 58.0% -- replaced with /var/log/ceph/cluster2-client.mirror.29403.log.gz 2024-08-31T22:55:55.806 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101510.log.gz 2024-08-31T22:55:55.806 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115683.log.gz 2024-08-31T22:55:55.807 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.148956.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59824.log.gz 2024-08-31T22:55:55.807 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.148956.log.gz 2024-08-31T22:55:55.808 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39591.log 2024-08-31T22:55:55.808 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.126609.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164658.log 2024-08-31T22:55:55.808 INFO:teuthology.orchestra.run.smithi032.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62703.log.gz 2024-08-31T22:55:55.808 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39591.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82982.log 2024-08-31T22:55:55.809 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164658.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.164658.log.gz 2024-08-31T22:55:55.809 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44476.log 2024-08-31T22:55:55.810 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82982.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74459.log 2024-08-31T22:55:55.810 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.44476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162076.log 2024-08-31T22:55:55.813 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.74459.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74742.log.gz 2024-08-31T22:55:55.815 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68796.log 2024-08-31T22:55:55.815 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162076.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.162076.log.gz 2024-08-31T22:55:55.817 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.79114.log.gz 2024-08-31T22:55:55.820 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67021.log.gz 2024-08-31T22:55:55.823 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161491.log 2024-08-31T22:55:55.823 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.68796.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141007.log 2024-08-31T22:55:55.824 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161491.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.161491.log.gz 2024-08-31T22:55:55.825 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137436.log 2024-08-31T22:55:55.826 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.141007.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.141007.log.gz 2024-08-31T22:55:55.826 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106407.log 2024-08-31T22:55:55.827 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.137436.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.97367.log.gz 2024-08-31T22:55:55.827 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.137436.log.gz 2024-08-31T22:55:55.827 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33297.log 2024-08-31T22:55:55.828 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.106407.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.80924.log.gz 2024-08-31T22:55:55.828 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77657.log 2024-08-31T22:55:55.829 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33297.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.47373.log.gz 2024-08-31T22:55:55.829 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163077.log 2024-08-31T22:55:55.829 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77657.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.47501.log.gz 2024-08-31T22:55:55.831 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.154569.log 2024-08-31T22:55:55.832 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163077.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.163077.log.gz 2024-08-31T22:55:55.836 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33669.log.gz 2024-08-31T22:55:55.836 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51511.log.gz 2024-08-31T22:55:55.836 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52034.log.gz 2024-08-31T22:55:55.837 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51321.log 2024-08-31T22:55:55.838 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.154569.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.154569.log.gz 2024-08-31T22:55:55.838 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120276.log 2024-08-31T22:55:55.839 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51321.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157279.log 2024-08-31T22:55:55.839 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.120276.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110968.log 2024-08-31T22:55:55.842 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157279.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.157279.log.gz 2024-08-31T22:55:55.842 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57452.log 2024-08-31T22:55:55.843 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.110968.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52220.log.gz 2024-08-31T22:55:55.847 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52689.log.gz 2024-08-31T22:55:55.850 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.77657.log.gz 2024-08-31T22:55:55.850 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55305.log.gz 2024-08-31T22:55:55.850 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33297.log.gz 2024-08-31T22:55:55.851 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132265.log 2024-08-31T22:55:55.852 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.57452.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157259.log 2024-08-31T22:55:55.853 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.132265.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.132265.log.gz 2024-08-31T22:55:55.853 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40335.log 2024-08-31T22:55:55.854 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157259.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.157259.log.gz 2024-08-31T22:55:55.854 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166527.log 2024-08-31T22:55:55.855 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.40335.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48342.log 2024-08-31T22:55:55.856 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.166527.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.93569.log.gz 2024-08-31T22:55:55.857 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166527.log.gz 2024-08-31T22:55:55.857 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109806.log.gz 2024-08-31T22:55:55.857 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.110178.log.gz 2024-08-31T22:55:55.859 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129648.log 2024-08-31T22:55:55.859 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48342.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57208.log 2024-08-31T22:55:55.860 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.111465.log.gz 2024-08-31T22:55:55.860 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.129648.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129648.log.gz 2024-08-31T22:55:55.861 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.86342.log.gz 2024-08-31T22:55:55.861 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.78370.log.gz 2024-08-31T22:55:55.862 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75599.log.gz 2024-08-31T22:55:55.862 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.89892.log.gz 2024-08-31T22:55:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125921.log 2024-08-31T22:55:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.57208.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56646.log 2024-08-31T22:55:55.869 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.125921.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.125921.log.gz 2024-08-31T22:55:55.869 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50442.log 2024-08-31T22:55:55.875 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.56646.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113616.log 2024-08-31T22:55:55.875 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.50442.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113426.log 2024-08-31T22:55:55.888 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.113616.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.90043.log.gz 2024-08-31T22:55:55.888 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.90171.log.gz 2024-08-31T22:55:55.888 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38901.log 2024-08-31T22:55:55.888 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.113426.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42980.log.gz 2024-08-31T22:55:55.889 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71789.log 2024-08-31T22:55:55.889 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.38901.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57208.log.gz 2024-08-31T22:55:55.889 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105380.log 2024-08-31T22:55:55.889 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57452.log.gz 2024-08-31T22:55:55.889 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63754.log.gz 2024-08-31T22:55:55.890 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.86776.log.gz 2024-08-31T22:55:55.890 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.90770.log.gz 2024-08-31T22:55:55.890 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.71789.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89109.log 2024-08-31T22:55:55.891 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105380.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47036.log 2024-08-31T22:55:55.891 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68796.log.gz 2024-08-31T22:55:55.891 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66986.log.gz 2024-08-31T22:55:55.893 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89109.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159352.log 2024-08-31T22:55:55.893 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.47036.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53189.log 2024-08-31T22:55:55.893 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71789.log.gz 2024-08-31T22:55:55.894 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106407.log.gz 2024-08-31T22:55:55.894 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159352.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44476.log.gz 2024-08-31T22:55:55.895 INFO:teuthology.orchestra.run.smithi032.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.mirror.0.29552.log.gz 2024-08-31T22:55:55.895 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.110968.log.gz 2024-08-31T22:55:55.897 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39591.log.gz 2024-08-31T22:55:55.897 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.50442.log.gz 2024-08-31T22:55:55.897 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101382.log.gz 2024-08-31T22:55:55.897 INFO:teuthology.orchestra.run.smithi032.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.159352.log.gz 2024-08-31T22:55:55.897 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145925.log 2024-08-31T22:55:55.897 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53189.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110085.log 2024-08-31T22:55:55.898 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.145925.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113426.log.gz 2024-08-31T22:55:55.899 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.113616.log.gz 2024-08-31T22:55:55.899 INFO:teuthology.orchestra.run.smithi032.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.145925.log.gz 2024-08-31T22:55:55.900 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105380.log.gz 2024-08-31T22:55:55.901 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.117144.log.gz 2024-08-31T22:55:55.902 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.117179.log.gz 2024-08-31T22:55:55.902 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40428.log 2024-08-31T22:55:55.903 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.110085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.166104.log 2024-08-31T22:55:55.903 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.117822.log.gz 2024-08-31T22:55:55.903 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.40428.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56646.log.gz 2024-08-31T22:55:55.905 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.58049.log.gz 2024-08-31T22:55:55.907 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107058.log 2024-08-31T22:55:55.907 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.166104.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.166104.log.gz 2024-08-31T22:55:55.908 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35816.log 2024-08-31T22:55:55.909 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107058.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67583.log 2024-08-31T22:55:55.910 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.35816.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123450.log 2024-08-31T22:55:55.911 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67583.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91505.log 2024-08-31T22:55:55.913 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.123450.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.123042.log.gz 2024-08-31T22:55:55.914 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138978.log 2024-08-31T22:55:55.914 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.91505.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72688.log 2024-08-31T22:55:55.916 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.138978.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47652.log.gz 2024-08-31T22:55:55.917 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47935.log.gz 2024-08-31T22:55:55.918 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48776.log.gz 2024-08-31T22:55:55.919 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.49241.log.gz 2024-08-31T22:55:55.920 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.49745.log.gz 2024-08-31T22:55:55.920 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49838.log.gz 2024-08-31T22:55:55.921 INFO:teuthology.orchestra.run.smithi032.stderr: 96.7% -- replaced with /var/log/ceph/cluster2-client.mirror.138978.log.gz 2024-08-31T22:55:55.922 INFO:teuthology.orchestra.run.smithi007.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-08-31T22:55:55.922 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51790.log 2024-08-31T22:55:55.923 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.72688.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.89109.log.gz 2024-08-31T22:55:55.923 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85373.log 2024-08-31T22:55:55.923 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35816.log.gz 2024-08-31T22:55:55.923 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40335.log.gz 2024-08-31T22:55:55.923 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51790.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40428.log.gz 2024-08-31T22:55:55.924 INFO:teuthology.orchestra.run.smithi032.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.118748.log.gz 2024-08-31T22:55:55.924 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83133.log 2024-08-31T22:55:55.925 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.85373.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36785.log 2024-08-31T22:55:55.926 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83133.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156186.log 2024-08-31T22:55:55.926 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.36785.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56611.log 2024-08-31T22:55:55.928 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156186.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.156186.log.gz 2024-08-31T22:55:55.928 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161288.log 2024-08-31T22:55:55.929 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.56611.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79804.log 2024-08-31T22:55:55.930 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161288.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.161288.log.gz 2024-08-31T22:55:55.930 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127076.log 2024-08-31T22:55:55.931 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.79804.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118771.log 2024-08-31T22:55:55.932 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.127076.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.127076.log.gz 2024-08-31T22:55:55.932 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82234.log 2024-08-31T22:55:55.933 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148570.log 2024-08-31T22:55:55.933 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82234.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76874.log 2024-08-31T22:55:55.935 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.148570.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148570.log.gz 2024-08-31T22:55:55.939 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159210.log 2024-08-31T22:55:55.939 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.76874.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138258.log 2024-08-31T22:55:55.942 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159210.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.159210.log.gz 2024-08-31T22:55:55.943 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149001.log 2024-08-31T22:55:55.943 INFO:teuthology.orchestra.run.smithi032.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.165718.log.gz 2024-08-31T22:55:55.944 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.138258.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.138258.log.gz 2024-08-31T22:55:55.944 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107058.log.gz 2024-08-31T22:55:55.944 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118431.log 2024-08-31T22:55:55.945 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48063.log 2024-08-31T22:55:55.945 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110085.log.gz 2024-08-31T22:55:55.945 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.118431.log: /var/log/ceph/cluster1-client.mirror.149001.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111748.log 2024-08-31T22:55:55.946 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48063.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.118208.log.gz 2024-08-31T22:55:55.947 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162188.log 2024-08-31T22:55:55.947 INFO:teuthology.orchestra.run.smithi032.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149001.log.gz 2024-08-31T22:55:55.947 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111992.log.gz 2024-08-31T22:55:55.947 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93255.log 2024-08-31T22:55:55.948 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.111748.log: /var/log/ceph/cluster1-client.mirror.162188.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.162188.log.gz 2024-08-31T22:55:55.948 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87520.log 2024-08-31T22:55:55.949 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.93255.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70630.log.gz 2024-08-31T22:55:55.949 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.154955.log 2024-08-31T22:55:55.949 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.87520.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82792.log 2024-08-31T22:55:55.950 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.154955.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.154955.log.gz 2024-08-31T22:55:55.950 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118881.log 2024-08-31T22:55:55.951 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82792.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112027.log 2024-08-31T22:55:55.951 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.118881.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67490.log 2024-08-31T22:55:55.952 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.97960.log.gz 2024-08-31T22:55:55.955 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.112027.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44511.log 2024-08-31T22:55:55.955 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67490.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70913.log 2024-08-31T22:55:55.956 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.44511.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119147.log 2024-08-31T22:55:55.956 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.70913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37440.log 2024-08-31T22:55:55.957 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119147.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70106.log 2024-08-31T22:55:55.957 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.37440.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78087.log 2024-08-31T22:55:55.958 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.70106.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81114.log 2024-08-31T22:55:55.959 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.78087.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65901.log 2024-08-31T22:55:55.959 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.81114.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79990.log 2024-08-31T22:55:55.960 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65901.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35258.log 2024-08-31T22:55:55.960 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.79990.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91852.log 2024-08-31T22:55:55.961 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.35258.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36657.log 2024-08-31T22:55:55.962 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.91852.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33146.log 2024-08-31T22:55:55.962 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.36657.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63064.log 2024-08-31T22:55:55.963 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42178.log 2024-08-31T22:55:55.963 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63064.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37967.log 2024-08-31T22:55:55.967 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.42178.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165367.log 2024-08-31T22:55:55.967 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.37967.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136114.log 2024-08-31T22:55:55.967 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96712.log 2024-08-31T22:55:55.968 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165367.log: /var/log/ceph/cluster2-client.mirror.136114.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.165367.log.gz 2024-08-31T22:55:55.969 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101045.log 2024-08-31T22:55:55.969 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96712.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62440.log 2024-08-31T22:55:55.970 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101045.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125490.log 2024-08-31T22:55:55.971 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62440.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158775.log 2024-08-31T22:55:55.978 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.125490.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.79711.log.gz 2024-08-31T22:55:55.979 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163151.log 2024-08-31T22:55:55.979 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65901.log.gz 2024-08-31T22:55:55.985 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158775.log: 90.7% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69079.log.gz 2024-08-31T22:55:55.985 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67583.log.gz 2024-08-31T22:55:55.985 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.158775.log.gz 2024-08-31T22:55:55.988 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74459.log.gz 2024-08-31T22:55:55.988 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.79990.log.gz 2024-08-31T22:55:55.988 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101045.log.gz 2024-08-31T22:55:55.989 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97495.log 2024-08-31T22:55:55.989 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163151.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.102502.log.gz 2024-08-31T22:55:55.989 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.163151.log.gz 2024-08-31T22:55:55.990 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37440.log.gz 2024-08-31T22:55:55.990 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.94751.log.gz 2024-08-31T22:55:55.991 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.37967.log.gz 2024-08-31T22:55:55.993 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.82792.log.gz 2024-08-31T22:55:55.994 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48342.log.gz 2024-08-31T22:55:55.995 INFO:teuthology.orchestra.run.smithi032.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.146345.log.gz 2024-08-31T22:55:55.995 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.84532.log.gz 2024-08-31T22:55:55.996 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80889.log 2024-08-31T22:55:55.996 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.97495.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111372.log 2024-08-31T22:55:55.997 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.80889.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160726.log 2024-08-31T22:55:55.998 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.111372.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103436.log 2024-08-31T22:55:55.999 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.160726.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.160726.log.gz 2024-08-31T22:55:56.003 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.162952.log 2024-08-31T22:55:56.003 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.103436.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119930.log 2024-08-31T22:55:56.004 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.162952.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.162952.log.gz 2024-08-31T22:55:56.009 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37219.log.gz 2024-08-31T22:55:56.011 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.43263.log.gz 2024-08-31T22:55:56.011 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43449.log.gz 2024-08-31T22:55:56.011 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43391.log.gz 2024-08-31T22:55:56.012 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43728.log.gz 2024-08-31T22:55:56.014 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.153046.log.gz 2024-08-31T22:55:56.014 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88024.log 2024-08-31T22:55:56.015 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119930.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101696.log 2024-08-31T22:55:56.016 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35165.log 2024-08-31T22:55:56.016 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.88024.log: /var/log/ceph/cluster2-client.mirror.101696.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62903.log 2024-08-31T22:55:56.017 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.35165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115189.log 2024-08-31T22:55:56.018 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62903.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149267.log 2024-08-31T22:55:56.018 INFO:teuthology.orchestra.run.smithi032.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118458.log.gz 2024-08-31T22:55:56.019 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.115189.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.155869.log 2024-08-31T22:55:56.019 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.149267.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113298.log 2024-08-31T22:55:56.019 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.149267.log.gz 2024-08-31T22:55:56.020 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.155869.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153811.log 2024-08-31T22:55:56.021 INFO:teuthology.orchestra.run.smithi032.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.155869.log.gz 2024-08-31T22:55:56.023 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.113298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161446.log 2024-08-31T22:55:56.024 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84660.log 2024-08-31T22:55:56.024 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.153811.log: /var/log/ceph/cluster1-client.mirror.161446.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.161446.log.gz 2024-08-31T22:55:56.024 INFO:teuthology.orchestra.run.smithi032.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.163280.log.gz 2024-08-31T22:55:56.028 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164106.log 2024-08-31T22:55:56.028 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.84660.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166371.log 2024-08-31T22:55:56.029 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105659.log 2024-08-31T22:55:56.029 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164106.log: /var/log/ceph/cluster2-client.admin.166371.log: 0.0% 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.164106.log.gz 2024-08-31T22:55:56.030 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.166371.log.gz 2024-08-31T22:55:56.030 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160304.log 2024-08-31T22:55:56.030 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159310.log 2024-08-31T22:55:56.030 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105659.log: /var/log/ceph/cluster2-client.mirror.160304.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86493.log 2024-08-31T22:55:56.032 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159310.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.160304.log.gz 2024-08-31T22:55:56.032 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.97495.log.gz 2024-08-31T22:55:56.033 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59599.log 2024-08-31T22:55:56.033 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86493.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.159310.log.gz 2024-08-31T22:55:56.033 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61374.log 2024-08-31T22:55:56.033 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.111748.log.gz 2024-08-31T22:55:56.033 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.59599.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144217.log 2024-08-31T22:55:56.034 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99363.log 2024-08-31T22:55:56.034 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61374.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83598.log 2024-08-31T22:55:56.034 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.144217.log: /var/log/ceph/cluster2-client.mirror.99363.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.112027.log.gz 2024-08-31T22:55:56.035 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.165958.log 2024-08-31T22:55:56.035 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83598.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.81114.log.gz 2024-08-31T22:55:56.036 INFO:teuthology.orchestra.run.smithi032.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.144217.log.gz 2024-08-31T22:55:56.036 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118098.log 2024-08-31T22:55:56.037 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.165958.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.165958.log.gz 2024-08-31T22:55:56.037 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57301.log 2024-08-31T22:55:56.037 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95123.log 2024-08-31T22:55:56.038 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118098.log: /var/log/ceph/cluster2-client.mirror.57301.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93666.log 2024-08-31T22:55:56.039 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160071.log 2024-08-31T22:55:56.039 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.95123.log: /var/log/ceph/cluster2-client.mirror.93666.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35258.log.gz 2024-08-31T22:55:56.042 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111558.log 2024-08-31T22:55:56.043 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.160071.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28295.log 2024-08-31T22:55:56.043 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.160071.log.gz/var/log/ceph/cluster2-client.mirror.111558.log: 2024-08-31T22:55:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126586.log 2024-08-31T22:55:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79897.log 2024-08-31T22:55:56.045 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28295.log: /var/log/ceph/cluster2-client.mirror.126586.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28295.log.gz 2024-08-31T22:55:56.045 INFO:teuthology.orchestra.run.smithi032.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.126586.log.gz 2024-08-31T22:55:56.045 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88493.log 2024-08-31T22:55:56.045 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.79897.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40990.log 2024-08-31T22:55:56.046 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49334.log 2024-08-31T22:55:56.046 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.88493.log: /var/log/ceph/cluster2-client.mirror.40990.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47222.log 2024-08-31T22:55:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72874.log 2024-08-31T22:55:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49334.log: /var/log/ceph/cluster2-client.mirror.47222.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81951.log 2024-08-31T22:55:56.048 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45819.log 2024-08-31T22:55:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.72874.log: /var/log/ceph/cluster2-client.mirror.81951.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34824.log 2024-08-31T22:55:56.050 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72967.log 2024-08-31T22:55:56.050 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.45819.log: /var/log/ceph/cluster2-client.mirror.34824.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90667.log 2024-08-31T22:55:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105190.log 2024-08-31T22:55:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.72967.log: /var/log/ceph/cluster1-client.admin.90667.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38432.log 2024-08-31T22:55:56.052 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.164299.log 2024-08-31T22:55:56.052 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105190.log: /var/log/ceph/cluster2-client.mirror.38432.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66335.log 2024-08-31T22:55:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162683.log 2024-08-31T22:55:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.164299.log: /var/log/ceph/cluster2-client.mirror.66335.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.164299.log.gz 2024-08-31T22:55:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82699.log 2024-08-31T22:55:56.054 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72591.log 2024-08-31T22:55:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162683.log: /var/log/ceph/cluster2-client.mirror.82699.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.162683.log.gz 2024-08-31T22:55:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164125.log 2024-08-31T22:55:56.056 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.72591.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141893.log 2024-08-31T22:55:56.056 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164125.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.164125.log.gz 2024-08-31T22:55:56.057 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54220.log 2024-08-31T22:55:56.059 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.141893.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.141893.log.gz 2024-08-31T22:55:56.060 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.79804.log.gz 2024-08-31T22:55:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109430.log 2024-08-31T22:55:56.062 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112647.log 2024-08-31T22:55:56.062 INFO:teuthology.orchestra.run.smithi032.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.123450.log.gz 2024-08-31T22:55:56.062 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.54220.log: /var/log/ceph/cluster2-client.mirror.109430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62591.log 2024-08-31T22:55:56.063 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.112647.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104667.log 2024-08-31T22:55:56.063 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.79897.log.gz 2024-08-31T22:55:56.063 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88024.log.gz 2024-08-31T22:55:56.063 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92790.log 2024-08-31T22:55:56.063 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62591.log: /var/log/ceph/cluster2-client.mirror.104667.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62347.log 2024-08-31T22:55:56.063 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61374.log.gz 2024-08-31T22:55:56.065 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.92790.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156228.log 2024-08-31T22:55:56.065 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.91852.log.gz 2024-08-31T22:55:56.066 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62347.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76219.log 2024-08-31T22:55:56.066 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156228.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85501.log 2024-08-31T22:55:56.066 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28699.log 2024-08-31T22:55:56.067 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.76219.log: /var/log/ceph/cluster2-client.mirror.85501.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79397.log 2024-08-31T22:55:56.067 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.28699.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.156228.log.gz 2024-08-31T22:55:56.067 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28699.log.gz 2024-08-31T22:55:56.067 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96712.log.gz 2024-08-31T22:55:56.068 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67490.log.gz 2024-08-31T22:55:56.068 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51569.log 2024-08-31T22:55:56.069 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.79397.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78498.log 2024-08-31T22:55:56.069 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92693.log 2024-08-31T22:55:56.069 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72874.log.gz 2024-08-31T22:55:56.070 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.78498.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141074.log 2024-08-31T22:55:56.070 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.92693.log: -- /var/log/ceph/cluster1-client.mirror.164332.log 2024-08-31T22:55:56.070 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.80889.log.gz 2024-08-31T22:55:56.070 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.141074.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.82982.log.gz 2024-08-31T22:55:56.070 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.83226.log.gz 2024-08-31T22:55:56.071 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.104667.log.gz 2024-08-31T22:55:56.073 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44511.log.gz 2024-08-31T22:55:56.073 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99363.log.gz 2024-08-31T22:55:56.073 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111372.log.gz 2024-08-31T22:55:56.075 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104574.log 2024-08-31T22:55:56.075 INFO:teuthology.orchestra.run.smithi032.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141074.log.gz 2024-08-31T22:55:56.075 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.164332.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.112647.log.gz 2024-08-31T22:55:56.075 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119210.log 2024-08-31T22:55:56.076 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100576.log 2024-08-31T22:55:56.076 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.164332.log.gz 2024-08-31T22:55:56.077 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59599.log.gz 2024-08-31T22:55:56.078 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105659.log.gz 2024-08-31T22:55:56.078 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47036.log.gz 2024-08-31T22:55:56.078 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119210.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38901.log.gz 2024-08-31T22:55:56.078 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.104574.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.90667.log.gz 2024-08-31T22:55:56.078 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158249.log 2024-08-31T22:55:56.079 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80331.log 2024-08-31T22:55:56.079 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.100576.log: /var/log/ceph/cluster2-client.mirror.158249.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96933.log 2024-08-31T22:55:56.079 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47466.log 2024-08-31T22:55:56.079 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48063.log.gz 2024-08-31T22:55:56.080 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.80331.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.51569.log.gz 2024-08-31T22:55:56.080 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164374.log 2024-08-31T22:55:56.080 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96933.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.158249.log.gz 2024-08-31T22:55:56.080 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.47466.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44755.log 2024-08-31T22:55:56.081 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164374.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.78087.log.gz 2024-08-31T22:55:56.081 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.164374.log.gz 2024-08-31T22:55:56.081 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56611.log.gz 2024-08-31T22:55:56.082 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36657.log.gz 2024-08-31T22:55:56.082 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68048.log 2024-08-31T22:55:56.082 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137308.log 2024-08-31T22:55:56.083 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.68048.log: /var/log/ceph/cluster2-client.mirror.44755.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103677.log 2024-08-31T22:55:56.083 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.137308.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45317.log 2024-08-31T22:55:56.083 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.103677.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.137308.log.gz 2024-08-31T22:55:56.083 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89857.log 2024-08-31T22:55:56.085 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.45317.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154175.log 2024-08-31T22:55:56.085 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89857.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71320.log 2024-08-31T22:55:56.085 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.154175.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62215.log 2024-08-31T22:55:56.085 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.71320.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116303.log 2024-08-31T22:55:56.086 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147010.log 2024-08-31T22:55:56.086 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62215.log: /var/log/ceph/cluster2-client.mirror.116303.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156956.log 2024-08-31T22:55:56.087 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.147010.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154175.log.gz 2024-08-31T22:55:56.087 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.147010.log.gz 2024-08-31T22:55:56.087 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111241.log 2024-08-31T22:55:56.088 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156956.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118411.log 2024-08-31T22:55:56.088 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.111241.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.156956.log.gz 2024-08-31T22:55:56.088 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164951.log 2024-08-31T22:55:56.089 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118411.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68854.log 2024-08-31T22:55:56.089 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.109430.log.gz 2024-08-31T22:55:56.089 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164951.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62772.log 2024-08-31T22:55:56.090 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.68854.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163172.log 2024-08-31T22:55:56.090 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.164951.log.gz 2024-08-31T22:55:56.090 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62772.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164456.log 2024-08-31T22:55:56.091 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163172.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101754.log 2024-08-31T22:55:56.091 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164456.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.163172.log.gz 2024-08-31T22:55:56.091 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.164456.log.gz 2024-08-31T22:55:56.091 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99642.log 2024-08-31T22:55:56.091 INFO:teuthology.orchestra.run.smithi032.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118055.log.gz 2024-08-31T22:55:56.092 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101754.log: 88.4%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.78498.log.gz -5 2024-08-31T22:55:56.092 INFO:teuthology.orchestra.run.smithi032.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.76405.log 2024-08-31T22:55:56.092 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.99642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162278.log 2024-08-31T22:55:56.093 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144239.log 2024-08-31T22:55:56.093 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162278.log: /var/log/ceph/cluster2-client.mirror.76405.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108275.log 2024-08-31T22:55:56.093 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.162278.log.gz 2024-08-31T22:55:56.094 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.144239.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35165.log.gz 2024-08-31T22:55:56.094 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76874.log.gz 2024-08-31T22:55:56.095 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85373.log.gz 2024-08-31T22:55:56.095 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.92693.log.gz 2024-08-31T22:55:56.096 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.80331.log.gz 2024-08-31T22:55:56.096 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.83598.log.gz 2024-08-31T22:55:56.096 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.95123.log.gz 2024-08-31T22:55:56.097 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.96933.log.gz 2024-08-31T22:55:56.099 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93255.log.gz 2024-08-31T22:55:56.103 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113391.log 2024-08-31T22:55:56.103 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65401.log 2024-08-31T22:55:56.105 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.108275.log: /var/log/ceph/cluster2-client.mirror.113391.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94879.log 2024-08-31T22:55:56.105 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67676.log 2024-08-31T22:55:56.106 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65401.log: /var/log/ceph/cluster2-client.mirror.94879.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113484.log 2024-08-31T22:55:56.106 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99793.log 2024-08-31T22:55:56.107 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67676.log: /var/log/ceph/cluster2-client.mirror.113484.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73994.log 2024-08-31T22:55:56.107 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.99793.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158897.log 2024-08-31T22:55:56.108 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148981.log 2024-08-31T22:55:56.108 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.73994.log: /var/log/ceph/cluster2-client.mirror.158897.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.158897.log.gz 2024-08-31T22:55:56.109 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41208.log 2024-08-31T22:55:56.111 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.148981.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.148981.log.gz 2024-08-31T22:55:56.111 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41279.log 2024-08-31T22:55:56.112 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.41208.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135815.log 2024-08-31T22:55:56.112 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.41279.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33483.log 2024-08-31T22:55:56.115 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.135815.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.135815.log.gz 2024-08-31T22:55:56.115 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109116.log 2024-08-31T22:55:56.117 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33483.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.118658.log.gz 2024-08-31T22:55:56.117 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138099.log 2024-08-31T22:55:56.117 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.79397.log.gz 2024-08-31T22:55:56.118 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.109116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52910.log 2024-08-31T22:55:56.119 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.138099.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138099.log.gz 2024-08-31T22:55:56.120 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150458.log 2024-08-31T22:55:56.120 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51321.log.gz 2024-08-31T22:55:56.121 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.150458.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137343.log 2024-08-31T22:55:56.121 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.52910.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51790.log.gz 2024-08-31T22:55:56.121 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54592.log 2024-08-31T22:55:56.122 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.88493.log.gz 2024-08-31T22:55:56.122 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.137343.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.137343.log.gz 2024-08-31T22:55:56.123 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45819.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.90898.log.gz 2024-08-31T22:55:56.123 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:56.123 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54220.log.gz 2024-08-31T22:55:56.123 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68854.log.gz 2024-08-31T22:55:56.124 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36975.log 2024-08-31T22:55:56.124 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.54592.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133486.log 2024-08-31T22:55:56.125 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.36975.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126415.log 2024-08-31T22:55:56.126 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.133486.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133486.log.gz 2024-08-31T22:55:56.126 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70913.log.gz 2024-08-31T22:55:56.128 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44383.log 2024-08-31T22:55:56.130 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.126415.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82234.log.gz 2024-08-31T22:55:56.134 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99642.log.gz 2024-08-31T22:55:56.139 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.84660.log.gz 2024-08-31T22:55:56.139 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.82699.log.gz 2024-08-31T22:55:56.140 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.111241.log.gz 2024-08-31T22:55:56.145 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47466.log.gz 2024-08-31T22:55:56.145 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.81951.log.gz 2024-08-31T22:55:56.145 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47222.log.gz 2024-08-31T22:55:56.146 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.113391.log.gz 2024-08-31T22:55:56.154 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105752.log 2024-08-31T22:55:56.161 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.44383.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36975.log.gz 2024-08-31T22:55:56.163 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159512.log 2024-08-31T22:55:56.165 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105752.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.118881.log.gz 2024-08-31T22:55:56.167 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.153811.log.gz 2024-08-31T22:55:56.167 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.163320.log 2024-08-31T22:55:56.167 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.108275.log.gz 2024-08-31T22:55:56.167 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109116.log.gz 2024-08-31T22:55:56.168 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159512.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.111558.log.gz 2024-08-31T22:55:56.168 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.159512.log.gz 2024-08-31T22:55:56.169 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96278.log 2024-08-31T22:55:56.169 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.163320.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.163320.log.gz 2024-08-31T22:55:56.170 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62122.log 2024-08-31T22:55:56.171 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96278.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118791.log 2024-08-31T22:55:56.171 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62122.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39742.log 2024-08-31T22:55:56.172 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.118791.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73095.log 2024-08-31T22:55:56.175 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39742.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36099.log 2024-08-31T22:55:56.176 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.73095.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48621.log 2024-08-31T22:55:56.177 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.36099.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48493.log 2024-08-31T22:55:56.177 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.89857.log.gz 2024-08-31T22:55:56.178 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48621.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161198.log 2024-08-31T22:55:56.179 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48493.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49183.log 2024-08-31T22:55:56.180 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161198.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.161198.log.gz 2024-08-31T22:55:56.180 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.165232.log 2024-08-31T22:55:56.181 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49183.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101851.log 2024-08-31T22:55:56.182 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.165232.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.165232.log.gz 2024-08-31T22:55:56.182 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160703.log 2024-08-31T22:55:56.183 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101851.log: 88.4%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.42178.log.gz 2024-08-31T22:55:56.183 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53658.log 2024-08-31T22:55:56.183 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76219.log.gz 2024-08-31T22:55:56.183 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.160703.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.160703.log.gz 2024-08-31T22:55:56.184 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49276.log 2024-08-31T22:55:56.185 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53658.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150825.log 2024-08-31T22:55:56.185 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76405.log.gz 2024-08-31T22:55:56.185 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49276.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85501.log.gz 2024-08-31T22:55:56.186 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63719.log 2024-08-31T22:55:56.188 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.150825.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.150825.log.gz 2024-08-31T22:55:56.189 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166475.log 2024-08-31T22:55:56.189 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63719.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71754.log 2024-08-31T22:55:56.190 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.166475.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166475.log.gz 2024-08-31T22:55:56.190 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55863.log 2024-08-31T22:55:56.191 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.71754.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140465.log 2024-08-31T22:55:56.191 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.55863.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146397.log 2024-08-31T22:55:56.192 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.140465.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62347.log.gz 2024-08-31T22:55:56.192 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62440.log.gz 2024-08-31T22:55:56.194 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62591.log.gz 2024-08-31T22:55:56.194 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.140465.log.gz 2024-08-31T22:55:56.194 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.125045.log.gz 2024-08-31T22:55:56.194 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107620.log 2024-08-31T22:55:56.194 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.146397.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91887.log 2024-08-31T22:55:56.194 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.92790.log.gz 2024-08-31T22:55:56.194 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.107620.log: -- replaced with /var/log/ceph/cluster2-client.mirror.146397.log.gz 2024-08-31T22:55:56.194 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110457.log 2024-08-31T22:55:56.195 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.91887.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.53189.log.gz 2024-08-31T22:55:56.195 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.83133.log.gz 2024-08-31T22:55:56.195 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.110457.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107655.log 2024-08-31T22:55:56.196 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70820.log 2024-08-31T22:55:56.196 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107655.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45317.log.gz 2024-08-31T22:55:56.198 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38432.log.gz 2024-08-31T22:55:56.199 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87896.log 2024-08-31T22:55:56.200 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141423.log 2024-08-31T22:55:56.200 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.70820.log: /var/log/ceph/cluster2-client.mirror.87896.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54592.log.gz 2024-08-31T22:55:56.201 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33983.log 2024-08-31T22:55:56.201 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72688.log.gz 2024-08-31T22:55:56.201 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.53658.log.gz 2024-08-31T22:55:56.201 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.141423.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62122.log.gz 2024-08-31T22:55:56.201 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.141423.log.gz 2024-08-31T22:55:56.202 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62215.log.gz 2024-08-31T22:55:56.202 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71320.log.gz 2024-08-31T22:55:56.202 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33983.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105132.log 2024-08-31T22:55:56.202 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113802.log 2024-08-31T22:55:56.203 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34979.log 2024-08-31T22:55:56.203 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72967.log.gz 2024-08-31T22:55:56.203 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.113802.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90991.log 2024-08-31T22:55:56.205 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.34979.log: /var/log/ceph/cluster2-client.mirror.90991.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119147.log.gz 2024-08-31T22:55:56.205 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119210.log.gz 2024-08-31T22:55:56.205 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118321.log 2024-08-31T22:55:56.206 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.101851.log.gz 2024-08-31T22:55:56.206 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.105190.log.gz 2024-08-31T22:55:56.206 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38618.log 2024-08-31T22:55:56.207 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118321.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157774.log 2024-08-31T22:55:56.207 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.38618.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73715.log 2024-08-31T22:55:56.209 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157774.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103436.log.gz 2024-08-31T22:55:56.210 INFO:teuthology.orchestra.run.smithi032.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.157774.log.gz 2024-08-31T22:55:56.210 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.103677.log.gz 2024-08-31T22:55:56.210 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46067.log 2024-08-31T22:55:56.210 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106721.log/var/log/ceph/cluster2-client.mirror.73715.log: 2024-08-31T22:55:56.211 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.104574.log.gz 2024-08-31T22:55:56.211 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45410.log 2024-08-31T22:55:56.211 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.46067.log: /var/log/ceph/cluster2-client.mirror.106721.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39742.log.gz 2024-08-31T22:55:56.212 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166128.log 2024-08-31T22:55:56.212 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.45410.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78684.log 2024-08-31T22:55:56.213 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.166128.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166128.log.gz 2024-08-31T22:55:56.213 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32491.log 2024-08-31T22:55:56.214 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.78684.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153769.log 2024-08-31T22:55:56.215 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.32491.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.101696.log.gz 2024-08-31T22:55:56.215 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.101754.log.gz 2024-08-31T22:55:56.215 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44383.log.gz 2024-08-31T22:55:56.215 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44755.log.gz 2024-08-31T22:55:56.215 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.99793.log.gz 2024-08-31T22:55:56.215 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105132.log.gz 2024-08-31T22:55:56.216 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100576.log.gz 2024-08-31T22:55:56.217 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.113298.log.gz 2024-08-31T22:55:56.217 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.116303.log.gz 2024-08-31T22:55:56.217 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.136114.log.gz 2024-08-31T22:55:56.218 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158733.log 2024-08-31T22:55:56.218 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.153769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.165170.log 2024-08-31T22:55:56.220 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158733.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.153769.log.gz 2024-08-31T22:55:56.221 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83040.log 2024-08-31T22:55:56.221 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.165170.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118681.log 2024-08-31T22:55:56.221 INFO:teuthology.orchestra.run.smithi032.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.118298.log.gz 2024-08-31T22:55:56.222 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83040.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.165170.log.gz 2024-08-31T22:55:56.222 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107620.log.gz 2024-08-31T22:55:56.222 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.107655.log.gz 2024-08-31T22:55:56.222 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37591.log 2024-08-31T22:55:56.222 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.113484.log.gz 2024-08-31T22:55:56.223 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110457.log.gz 2024-08-31T22:55:56.223 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118681.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33983.log.gz 2024-08-31T22:55:56.223 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67769.log 2024-08-31T22:55:56.223 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34979.log.gz 2024-08-31T22:55:56.223 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.78684.log.gz 2024-08-31T22:55:56.223 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87520.log.gz 2024-08-31T22:55:56.223 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.37591.log: 87.9% 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83040.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.34824.log.gz 2024-08-31T22:55:56.223 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:56.227 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.91887.log.gz 2024-08-31T22:55:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136811.log 2024-08-31T22:55:56.227 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87896.log.gz 2024-08-31T22:55:56.227 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.158733.log.gz 2024-08-31T22:55:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103006.log 2024-08-31T22:55:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67769.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65401.log.gz 2024-08-31T22:55:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112961.log 2024-08-31T22:55:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.136811.log: /var/log/ceph/cluster2-client.mirror.103006.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106314.log 2024-08-31T22:55:56.227 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163192.log 2024-08-31T22:55:56.227 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.136811.log.gz 2024-08-31T22:55:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.112961.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66335.log.gz 2024-08-31T22:55:56.227 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.106721.log.gz 2024-08-31T22:55:56.228 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116524.log 2024-08-31T22:55:56.228 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107151.log 2024-08-31T22:55:56.228 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163192.log: /var/log/ceph/cluster2-client.mirror.116524.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37591.log.gz 2024-08-31T22:55:56.228 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.106314.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58200.log 2024-08-31T22:55:56.228 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107151.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158917.log 2024-08-31T22:55:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58200.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159190.log 2024-08-31T22:55:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158917.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.163192.log.gz 2024-08-31T22:55:56.229 INFO:teuthology.orchestra.run.smithi032.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.158917.log.gz 2024-08-31T22:55:56.230 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36320.log 2024-08-31T22:55:56.231 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159190.log: gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-08-31T22:55:56.231 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.36320.log: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48307.log 2024-08-31T22:55:56.231 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.159190.log.gz 2024-08-31T22:55:56.231 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82757.log 2024-08-31T22:55:56.231 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2.audit.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.116710.log.gz 2024-08-31T22:55:56.232 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48307.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33611.log 2024-08-31T22:55:56.232 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82757.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159554.log 2024-08-31T22:55:56.232 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.162899.log 2024-08-31T22:55:56.234 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49055.log 2024-08-31T22:55:56.234 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159554.log: /var/log/ceph/cluster1-client.mirror.0.162899.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31708.log 2024-08-31T22:55:56.234 INFO:teuthology.orchestra.run.smithi032.stderr: 20.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.162899.log.gz 2024-08-31T22:55:56.234 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49055.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94410.log 2024-08-31T22:55:56.234 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.31708.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54778.log 2024-08-31T22:55:56.234 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94410.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109771.log 2024-08-31T22:55:56.235 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.54778.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49613.log 2024-08-31T22:55:56.236 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.109771.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38153.log 2024-08-31T22:55:56.236 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49613.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162593.log 2024-08-31T22:55:56.236 INFO:teuthology.orchestra.run.smithi032.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.159554.log.gz 2024-08-31T22:55:56.236 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.38153.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40207.log 2024-08-31T22:55:56.236 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162593.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73715.log.gz 2024-08-31T22:55:56.236 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.162593.log.gz 2024-08-31T22:55:56.237 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45503.log 2024-08-31T22:55:56.238 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.40207.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166579.log 2024-08-31T22:55:56.238 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.45503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131383.log 2024-08-31T22:55:56.238 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.166579.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110515.log 2024-08-31T22:55:56.238 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166579.log.gz 2024-08-31T22:55:56.238 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119059.log 2024-08-31T22:55:56.238 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.131383.log: /var/log/ceph/cluster2-client.mirror.110515.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45410.log.gz 2024-08-31T22:55:56.240 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132244.log 2024-08-31T22:55:56.240 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119059.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166501.log 2024-08-31T22:55:56.240 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.132244.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.162978.log 2024-08-31T22:55:56.240 INFO:teuthology.orchestra.run.smithi032.stderr: 2.6% -- replaced with /var/log/ceph/cluster2-client.mirror.132244.log.gz 2024-08-31T22:55:56.240 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.166501.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166501.log.gz 2024-08-31T22:55:56.240 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159048.log 2024-08-31T22:55:56.241 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.162978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139768.log 2024-08-31T22:55:56.241 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159048.log: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.126609.log.gz 2024-08-31T22:55:56.242 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.159048.log.gz 2024-08-31T22:55:56.242 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111841.log 2024-08-31T22:55:56.242 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80083.log 2024-08-31T22:55:56.242 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.139768.log: /var/log/ceph/cluster2-client.mirror.111841.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.139768.log.gz 2024-08-31T22:55:56.243 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32584.log 2024-08-31T22:55:56.243 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97402.log 2024-08-31T22:55:56.243 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.80083.log: /var/log/ceph/cluster2-client.mirror.32584.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98243.log 2024-08-31T22:55:56.244 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107341.log 2024-08-31T22:55:56.246 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.97402.log: /var/log/ceph/cluster2-client.mirror.98243.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52910.log.gz 2024-08-31T22:55:56.250 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70106.log.gz 2024-08-31T22:55:56.251 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164787.log 2024-08-31T22:55:56.251 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160681.log 2024-08-31T22:55:56.251 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164787.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53344.log 2024-08-31T22:55:56.252 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.160681.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.160681.log.gz 2024-08-31T22:55:56.253 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32491.log.gz 2024-08-31T22:55:56.261 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49648.log 2024-08-31T22:55:56.261 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36099.log.gz 2024-08-31T22:55:56.262 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36785.log.gz 2024-08-31T22:55:56.262 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36320.log.gz 2024-08-31T22:55:56.266 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53344.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38618.log.gz 2024-08-31T22:55:56.267 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77622.log 2024-08-31T22:55:56.267 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49648.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.163025.log 2024-08-31T22:55:56.268 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77622.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164482.log 2024-08-31T22:55:56.268 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.163025.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33483.log.gz 2024-08-31T22:55:56.269 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.163025.log.gz 2024-08-31T22:55:56.269 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118141.log 2024-08-31T22:55:56.275 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164482.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33146.log.gz 2024-08-31T22:55:56.277 INFO:teuthology.orchestra.run.smithi032.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118771.log.gz 2024-08-31T22:55:56.278 INFO:teuthology.orchestra.run.smithi032.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.150458.log.gz 2024-08-31T22:55:56.280 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40207.log.gz 2024-08-31T22:55:56.280 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60289.log 2024-08-31T22:55:56.281 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68013.log 2024-08-31T22:55:56.282 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60289.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75789.log 2024-08-31T22:55:56.282 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40990.log.gz 2024-08-31T22:55:56.282 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41208.log.gz 2024-08-31T22:55:56.282 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.68013.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93666.log.gz 2024-08-31T22:55:56.284 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116489.log 2024-08-31T22:55:56.284 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75789.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126354.log 2024-08-31T22:55:56.285 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.116489.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.94879.log.gz 2024-08-31T22:55:56.287 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.113802.log.gz 2024-08-31T22:55:56.287 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.98243.log.gz 2024-08-31T22:55:56.288 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79149.log 2024-08-31T22:55:56.289 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.126354.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50298.log.gz 2024-08-31T22:55:56.289 INFO:teuthology.orchestra.run.smithi032.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.126354.log.gz 2024-08-31T22:55:56.290 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41953.log 2024-08-31T22:55:56.290 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.79149.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45689.log 2024-08-31T22:55:56.291 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.41953.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87710.log 2024-08-31T22:55:56.292 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63064.log.gz 2024-08-31T22:55:56.292 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63719.log.gz 2024-08-31T22:55:56.292 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.90991.log.gz 2024-08-31T22:55:56.293 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67769.log.gz 2024-08-31T22:55:56.294 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70820.log.gz 2024-08-31T22:55:56.295 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.45689.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96433.log 2024-08-31T22:55:56.296 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.87710.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72591.log.gz 2024-08-31T22:55:56.296 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73095.log.gz 2024-08-31T22:55:56.296 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38153.log.gz 2024-08-31T22:55:56.297 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.110515.log.gz 2024-08-31T22:55:56.297 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103006.log.gz 2024-08-31T22:55:56.297 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.112961.log.gz 2024-08-31T22:55:56.297 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.116524.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.116489.log.gz 2024-08-31T22:55:56.298 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:56.298 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.82757.log.gz 2024-08-31T22:55:56.298 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153391.log 2024-08-31T22:55:56.299 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96433.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162391.log 2024-08-31T22:55:56.299 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.105752.log.gz 2024-08-31T22:55:56.301 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.153391.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.153391.log.gz 2024-08-31T22:55:56.301 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72068.log 2024-08-31T22:55:56.302 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162391.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.162391.log.gz 2024-08-31T22:55:56.302 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-08-31T22:55:56.303 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.72068.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122161.log 2024-08-31T22:55:56.304 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48621.log.gz 2024-08-31T22:55:56.305 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49055.log.gz 2024-08-31T22:55:56.306 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.49183.log.gz 2024-08-31T22:55:56.306 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49334.log.gz 2024-08-31T22:55:56.306 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49276.log.gz 2024-08-31T22:55:56.310 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/cluster2.log.gz 2024-08-31T22:55:56.312 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28574.log 2024-08-31T22:55:56.313 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.122161.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58200.log.gz 2024-08-31T22:55:56.327 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68389.log 2024-08-31T22:55:56.327 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.28574.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28574.log.gz 2024-08-31T22:55:56.338 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71506.log 2024-08-31T22:55:56.344 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.68389.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33014.log 2024-08-31T22:55:56.349 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.71506.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31708.log.gz 2024-08-31T22:55:56.351 INFO:teuthology.orchestra.run.smithi032.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118368.log.gz 2024-08-31T22:55:56.352 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.107341.log.gz 2024-08-31T22:55:56.354 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.109771.log.gz 2024-08-31T22:55:56.355 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71041.log 2024-08-31T22:55:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33014.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.144239.log.gz 2024-08-31T22:55:56.358 INFO:teuthology.orchestra.run.smithi032.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118028.log.gz 2024-08-31T22:55:56.358 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140629.log 2024-08-31T22:55:56.359 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.71041.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151091.log 2024-08-31T22:55:56.360 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.140629.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140629.log.gz 2024-08-31T22:55:56.360 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49520.log 2024-08-31T22:55:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.151091.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.151091.log.gz 2024-08-31T22:55:56.361 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33611.log.gz 2024-08-31T22:55:56.362 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.111841.log.gz 2024-08-31T22:55:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85877.log 2024-08-31T22:55:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49520.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75789.log.gz 2024-08-31T22:55:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35200.log 2024-08-31T22:55:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.85877.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114922.log 2024-08-31T22:55:56.363 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.87710.log.gz 2024-08-31T22:55:56.364 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.35200.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150416.log 2024-08-31T22:55:56.364 INFO:teuthology.orchestra.run.smithi032.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115189.log.gz 2024-08-31T22:55:56.365 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.114922.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45352.log 2024-08-31T22:55:56.366 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.150416.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.86493.log.gz 2024-08-31T22:55:56.366 INFO:teuthology.orchestra.run.smithi032.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.150416.log.gz 2024-08-31T22:55:56.369 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28139.log 2024-08-31T22:55:56.369 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.45352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45881.log 2024-08-31T22:55:56.372 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28139.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28139.log.gz 2024-08-31T22:55:56.373 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163527.log 2024-08-31T22:55:56.373 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.45881.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143626.log 2024-08-31T22:55:56.374 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163527.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.163527.log.gz 2024-08-31T22:55:56.375 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159583.log 2024-08-31T22:55:56.375 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.143626.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.143626.log.gz 2024-08-31T22:55:56.375 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98053.log 2024-08-31T22:55:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159583.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.159583.log.gz 2024-08-31T22:55:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38560.log 2024-08-31T22:55:56.380 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.98053.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164434.log 2024-08-31T22:55:56.381 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.38560.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57301.log.gz 2024-08-31T22:55:56.382 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45503.log.gz 2024-08-31T22:55:56.382 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45689.log.gz 2024-08-31T22:55:56.383 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45352.log.gz 2024-08-31T22:55:56.383 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42922.log 2024-08-31T22:55:56.383 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.54778.log.gz 2024-08-31T22:55:56.384 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164434.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.164434.log.gz 2024-08-31T22:55:56.384 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137735.log 2024-08-31T22:55:56.385 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67676.log.gz 2024-08-31T22:55:56.385 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.42922.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84288.log 2024-08-31T22:55:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.137735.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71041.log.gz 2024-08-31T22:55:56.387 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72068.log.gz 2024-08-31T22:55:56.389 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.80083.log.gz 2024-08-31T22:55:56.389 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106314.log.gz 2024-08-31T22:55:56.393 INFO:teuthology.orchestra.run.smithi032.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.125490.log.gz 2024-08-31T22:55:56.393 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68013.log.gz 2024-08-31T22:55:56.394 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68048.log.gz 2024-08-31T22:55:56.394 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144172.log 2024-08-31T22:55:56.394 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.84288.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55863.log.gz 2024-08-31T22:55:56.395 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161018.log 2024-08-31T22:55:56.395 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.144172.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49613.log.gz 2024-08-31T22:55:56.395 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49648.log.gz 2024-08-31T22:55:56.395 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.144172.log.gz 2024-08-31T22:55:56.396 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99514.log 2024-08-31T22:55:56.396 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161018.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.94410.log.gz 2024-08-31T22:55:56.397 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.84288.log.gz 2024-08-31T22:55:56.397 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.161018.log.gz 2024-08-31T22:55:56.397 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165415.log 2024-08-31T22:55:56.398 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48307.log.gz 2024-08-31T22:55:56.398 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.99514.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48493.log.gz 2024-08-31T22:55:56.398 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.49520.log.gz 2024-08-31T22:55:56.398 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156131.log 2024-08-31T22:55:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165415.log: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.126415.log.gz 2024-08-31T22:55:56.406 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42364.log 2024-08-31T22:55:56.408 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156131.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.156131.log.gz 2024-08-31T22:55:56.409 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100297.log 2024-08-31T22:55:56.409 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.42364.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92042.log 2024-08-31T22:55:56.410 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.100297.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47129.log 2024-08-31T22:55:56.411 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.92042.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57394.log 2024-08-31T22:55:56.412 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.47129.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55898.log 2024-08-31T22:55:56.413 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.57394.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80645.log 2024-08-31T22:55:56.414 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.55898.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97084.log 2024-08-31T22:55:56.415 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.80645.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123884.log 2024-08-31T22:55:56.415 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.97084.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165910.log 2024-08-31T22:55:56.417 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.123884.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41953.log.gz 2024-08-31T22:55:56.419 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.97402.log.gz 2024-08-31T22:55:56.419 INFO:teuthology.orchestra.run.smithi032.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77622.log.gz 2024-08-31T22:55:56.421 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.164482.log.gz 2024-08-31T22:55:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138293.log 2024-08-31T22:55:56.423 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165910.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.165910.log.gz 2024-08-31T22:55:56.423 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150745.log 2024-08-31T22:55:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.138293.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.138293.log.gz 2024-08-31T22:55:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51042.log 2024-08-31T22:55:56.425 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.150745.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32584.log.gz 2024-08-31T22:55:56.425 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33014.log.gz 2024-08-31T22:55:56.425 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.150745.log.gz 2024-08-31T22:55:56.425 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145581.log 2024-08-31T22:55:56.426 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51042.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108496.log 2024-08-31T22:55:56.426 INFO:teuthology.orchestra.run.smithi032.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.118431.log.gz 2024-08-31T22:55:56.426 INFO:teuthology.orchestra.run.smithi032.stderr: 93.3%/var/log/ceph/cluster2-client.mirror.145581.log: -- replaced with /var/log/ceph/cluster2-client.mirror.118411.log.gz 2024-08-31T22:55:56.428 INFO:teuthology.orchestra.run.smithi032.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.118791.log.gz 2024-08-31T22:55:56.439 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82048.log 2024-08-31T22:55:56.442 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.108496.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.107151.log.gz 2024-08-31T22:55:56.444 INFO:teuthology.orchestra.run.smithi032.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118098.log.gz 2024-08-31T22:55:56.447 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.108496.log.gz 2024-08-31T22:55:56.451 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.35200.log.gz 2024-08-31T22:55:56.456 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114922.log.gz 2024-08-31T22:55:56.457 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85877.log.gz 2024-08-31T22:55:56.457 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.98053.log.gz 2024-08-31T22:55:56.462 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75056.log 2024-08-31T22:55:56.471 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82048.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97925.log 2024-08-31T22:55:56.479 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75056.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98429.log 2024-08-31T22:55:56.485 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.97925.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57394.log.gz 2024-08-31T22:55:56.485 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.79149.log.gz 2024-08-31T22:55:56.487 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143831.log 2024-08-31T22:55:56.490 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.98429.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71754.log.gz 2024-08-31T22:55:56.490 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75056.log.gz 2024-08-31T22:55:56.491 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.137735.log.gz 2024-08-31T22:55:56.491 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30588.log 2024-08-31T22:55:56.493 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.143831.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.92042.log.gz 2024-08-31T22:55:56.494 INFO:teuthology.orchestra.run.smithi032.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.143831.log.gz 2024-08-31T22:55:56.494 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79769.log 2024-08-31T22:55:56.495 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.30588.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96336.log 2024-08-31T22:55:56.495 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.79769.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.155525.log 2024-08-31T22:55:56.496 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96336.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111713.log 2024-08-31T22:55:56.496 INFO:teuthology.orchestra.run.smithi032.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118321.log.gz 2024-08-31T22:55:56.500 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.155525.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42364.log.gz 2024-08-31T22:55:56.501 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.145581.log.gz 2024-08-31T22:55:56.502 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82385.log 2024-08-31T22:55:56.502 INFO:teuthology.orchestra.run.smithi032.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.155525.log.gz 2024-08-31T22:55:56.502 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.111713.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62312.log 2024-08-31T22:55:56.503 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82385.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145282.log 2024-08-31T22:55:56.503 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162301.log 2024-08-31T22:55:56.503 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.145282.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81486.log 2024-08-31T22:55:56.504 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162301.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.162301.log.gz 2024-08-31T22:55:56.505 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99514.log.gz 2024-08-31T22:55:56.505 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.145282.log.gz 2024-08-31T22:55:56.505 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152353.log 2024-08-31T22:55:56.505 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.81486.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35630.log 2024-08-31T22:55:56.506 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.152353.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29420.log 2024-08-31T22:55:56.507 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.35630.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.152353.log.gz 2024-08-31T22:55:56.508 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46718.log 2024-08-31T22:55:56.508 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.29420.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108089.log 2024-08-31T22:55:56.509 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.46718.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46009.log 2024-08-31T22:55:56.509 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.108089.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136293.log 2024-08-31T22:55:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138550.log 2024-08-31T22:55:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.46009.log: /var/log/ceph/cluster2-client.mirror.136293.log: /var/log/ceph/cluster2-client.mirror.138550.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.136293.log.gz 2024-08-31T22:55:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49989.log 2024-08-31T22:55:56.511 INFO:teuthology.orchestra.run.smithi032.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.138550.log.gz 2024-08-31T22:55:56.512 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29130.log 2024-08-31T22:55:56.512 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135872.log 2024-08-31T22:55:56.513 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49989.log: /var/log/ceph/cluster1-client.mirror.29130.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103877.log 2024-08-31T22:55:56.513 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56832.log 2024-08-31T22:55:56.514 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.135872.log: /var/log/ceph/cluster1-client.admin.103877.log: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.164787.log.gz 2024-08-31T22:55:56.515 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46718.log.gz 2024-08-31T22:55:56.515 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47129.log.gz 2024-08-31T22:55:56.515 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102537.log 2024-08-31T22:55:56.515 INFO:teuthology.orchestra.run.smithi032.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135872.log.gz 2024-08-31T22:55:56.516 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.56832.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48156.log 2024-08-31T22:55:56.516 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.102537.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77002.log 2024-08-31T22:55:56.516 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48156.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66649.log 2024-08-31T22:55:56.517 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79525.log 2024-08-31T22:55:56.517 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.66649.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64688.log 2024-08-31T22:55:56.517 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83075.log 2024-08-31T22:55:56.518 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64688.log: /var/log/ceph/cluster2-client.mirror.79525.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142319.log 2024-08-31T22:55:56.519 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83075.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111430.log 2024-08-31T22:55:56.519 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.142319.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96468.log 2024-08-31T22:55:56.519 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.111430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72533.log 2024-08-31T22:55:56.519 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.142319.log.gz 2024-08-31T22:55:56.519 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96468.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.82048.log.gz 2024-08-31T22:55:56.520 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73622.log 2024-08-31T22:55:56.521 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.72533.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51228.log 2024-08-31T22:55:56.521 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.73622.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41384.log 2024-08-31T22:55:56.522 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51228.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61002.log 2024-08-31T22:55:56.522 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.41384.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165692.log 2024-08-31T22:55:56.523 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161423.log 2024-08-31T22:55:56.523 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118251.log 2024-08-31T22:55:56.523 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161423.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.165692.log.gz 2024-08-31T22:55:56.523 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.161423.log.gz 2024-08-31T22:55:56.524 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71847.log 2024-08-31T22:55:56.524 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.118251.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65866.log 2024-08-31T22:55:56.525 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.71847.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39029.log 2024-08-31T22:55:56.525 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65866.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90956.log 2024-08-31T22:55:56.525 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162548.log 2024-08-31T22:55:56.526 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.90956.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107434.log 2024-08-31T22:55:56.526 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162548.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103610.log 2024-08-31T22:55:56.526 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.162548.log.gz 2024-08-31T22:55:56.527 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107434.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73560.log 2024-08-31T22:55:56.527 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.103610.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50363.log 2024-08-31T22:55:56.527 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.73560.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159281.log 2024-08-31T22:55:56.527 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50363.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92135.log 2024-08-31T22:55:56.527 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74777.log 2024-08-31T22:55:56.528 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159281.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55340.log 2024-08-31T22:55:56.528 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.92135.log: /var/log/ceph/cluster2-client.mirror.74777.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150847.log 2024-08-31T22:55:56.529 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.55340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77564.log 2024-08-31T22:55:56.529 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.150847.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.159281.log.gz 2024-08-31T22:55:56.537 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97026.log 2024-08-31T22:55:56.537 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77564.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120548.log 2024-08-31T22:55:56.538 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110798.log 2024-08-31T22:55:56.538 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.97026.log: /var/log/ceph/cluster2-client.mirror.120548.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108682.log 2024-08-31T22:55:56.539 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83505.log 2024-08-31T22:55:56.539 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.110798.log: /var/log/ceph/cluster2-client.mirror.108682.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70443.log 2024-08-31T22:55:56.540 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162031.log 2024-08-31T22:55:56.540 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83505.log: /var/log/ceph/cluster2-client.mirror.70443.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162865.log 2024-08-31T22:55:56.541 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40490.log 2024-08-31T22:55:56.541 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162031.log: /var/log/ceph/cluster1-client.mirror.162865.log: 54.0% -- replaced with /var/log/ceph/cluster1-client.mirror.162865.log.gz 2024-08-31T22:55:56.541 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.162031.log.gz 2024-08-31T22:55:56.542 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95964.log 2024-08-31T22:55:56.542 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104198.log 2024-08-31T22:55:56.542 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.40490.log: /var/log/ceph/cluster2-client.mirror.95964.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105938.log 2024-08-31T22:55:56.543 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157207.log 2024-08-31T22:55:56.544 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105938.log: /var/log/ceph/cluster2-client.mirror.104198.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47970.log 2024-08-31T22:55:56.544 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157207.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64653.log 2024-08-31T22:55:56.545 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.47970.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119319.log 2024-08-31T22:55:56.546 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64653.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31088.log 2024-08-31T22:55:56.546 INFO:teuthology.orchestra.run.smithi032.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.157207.log.gz 2024-08-31T22:55:56.547 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65153.log 2024-08-31T22:55:56.547 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119319.log: /var/log/ceph/cluster2-client.mirror.31088.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144197.log 2024-08-31T22:55:56.547 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93383.log 2024-08-31T22:55:56.548 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65153.log: /var/log/ceph/cluster2-client.mirror.144197.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157845.log 2024-08-31T22:55:56.549 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.93383.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165348.log 2024-08-31T22:55:56.549 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157845.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.144197.log.gz 2024-08-31T22:55:56.551 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29645.log 2024-08-31T22:55:56.551 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165348.log: 83.5% 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.157845.log.gz 2024-08-31T22:55:56.552 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.165348.log.gz 2024-08-31T22:55:56.552 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161063.log 2024-08-31T22:55:56.553 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78029.log 2024-08-31T22:55:56.553 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51042.log.gz 2024-08-31T22:55:56.553 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.29645.log: /var/log/ceph/cluster1-client.mirror.161063.log: 86.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64405.log 2024-08-31T22:55:56.553 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.161063.log.gz 2024-08-31T22:55:56.553 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53344.log.gz 2024-08-31T22:55:56.554 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.78029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117752.log 2024-08-31T22:55:56.554 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77750.log 2024-08-31T22:55:56.554 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64405.log: /var/log/ceph/cluster1-client.mirror.117752.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160233.log 2024-08-31T22:55:56.555 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38002.log 2024-08-31T22:55:56.555 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77750.log: /var/log/ceph/cluster2-client.mirror.160233.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.160233.log.gz 2024-08-31T22:55:56.555 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111149.log 2024-08-31T22:55:56.556 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30995.log 2024-08-31T22:55:56.557 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.38002.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35630.log.gz 2024-08-31T22:55:56.557 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.78029.log.gz 2024-08-31T22:55:56.558 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.111713.log.gz 2024-08-31T22:55:56.560 INFO:teuthology.orchestra.run.smithi032.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118681.log.gz 2024-08-31T22:55:56.560 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.111149.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98487.log 2024-08-31T22:55:56.561 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94561.log 2024-08-31T22:55:56.561 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.30995.log: /var/log/ceph/cluster2-client.mirror.98487.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119059.log.gz 2024-08-31T22:55:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85094.log 2024-08-31T22:55:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117915.log 2024-08-31T22:55:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94561.log: /var/log/ceph/cluster2-client.mirror.85094.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95309.log 2024-08-31T22:55:56.564 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.154920.log 2024-08-31T22:55:56.564 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.117915.log: /var/log/ceph/cluster2-client.mirror.95309.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98801.log 2024-08-31T22:55:56.565 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71506.log.gz 2024-08-31T22:55:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.154920.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.154920.log.gz 2024-08-31T22:55:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84625.log 2024-08-31T22:55:56.566 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37909.log 2024-08-31T22:55:56.571 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.98801.log: /var/log/ceph/cluster2-client.mirror.84625.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113523.log 2024-08-31T22:55:56.571 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.37909.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36134.log 2024-08-31T22:55:56.572 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.113523.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68296.log 2024-08-31T22:55:56.572 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.36134.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114980.log 2024-08-31T22:55:56.573 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35386.log 2024-08-31T22:55:56.573 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.68296.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42922.log.gz 2024-08-31T22:55:56.574 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.114980.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91362.log 2024-08-31T22:55:56.575 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.35386.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81579.log 2024-08-31T22:55:56.575 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.85094.log.gz 2024-08-31T22:55:56.575 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.91362.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.80645.log.gz 2024-08-31T22:55:56.576 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124166.log 2024-08-31T22:55:56.577 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.81579.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97925.log.gz 2024-08-31T22:55:56.577 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72909.log 2024-08-31T22:55:56.577 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.98429.log.gz 2024-08-31T22:55:56.577 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.124166.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.98487.log.gz 2024-08-31T22:55:56.578 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145668.log 2024-08-31T22:55:56.578 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73994.log.gz 2024-08-31T22:55:56.578 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.72909.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87241.log 2024-08-31T22:55:56.579 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.145668.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.145668.log.gz 2024-08-31T22:55:56.580 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114550.log 2024-08-31T22:55:56.580 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.87241.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165815.log 2024-08-31T22:55:56.581 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.114550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59169.log 2024-08-31T22:55:56.582 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165815.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38560.log.gz 2024-08-31T22:55:56.583 INFO:teuthology.orchestra.run.smithi032.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.165815.log.gz 2024-08-31T22:55:56.583 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.29540.log 2024-08-31T22:55:56.584 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.59169.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79211.log 2024-08-31T22:55:56.585 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.1.29540.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41709.log 2024-08-31T22:55:56.585 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.79211.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.162908.log 2024-08-31T22:55:56.586 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.41709.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96150.log 2024-08-31T22:55:56.587 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.96278.log.gz 2024-08-31T22:55:56.587 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.0.162908.log: 20.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.162908.log.gz 2024-08-31T22:55:56.590 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114139.log 2024-08-31T22:55:56.591 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.100297.log.gz 2024-08-31T22:55:56.591 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96150.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.79525.log.gz 2024-08-31T22:55:56.591 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.79769.log.gz 2024-08-31T22:55:56.591 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66649.log.gz 2024-08-31T22:55:56.592 INFO:teuthology.orchestra.run.smithi032.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.103765.log.gz 2024-08-31T22:55:56.592 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43112.log 2024-08-31T22:55:56.593 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.114139.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71940.log 2024-08-31T22:55:56.594 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.43112.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51755.log 2024-08-31T22:55:56.594 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.71940.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114267.log 2024-08-31T22:55:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51755.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64502.log 2024-08-31T22:55:56.596 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.114267.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60289.log.gz 2024-08-31T22:55:56.596 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53472.log 2024-08-31T22:55:56.596 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64502.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122684.log 2024-08-31T22:55:56.597 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53472.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103564.log 2024-08-31T22:55:56.598 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.122684.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.122684.log.gz 2024-08-31T22:55:56.598 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59355.log 2024-08-31T22:55:56.599 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.103564.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39180.log 2024-08-31T22:55:56.599 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.59355.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115015.log 2024-08-31T22:55:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39180.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62312.log.gz 2024-08-31T22:55:56.601 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.96433.log.gz 2024-08-31T22:55:56.602 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.96468.log.gz 2024-08-31T22:55:56.602 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72909.log.gz 2024-08-31T22:55:56.602 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37909.log.gz 2024-08-31T22:55:56.602 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71847.log.gz 2024-08-31T22:55:56.602 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.96336.log.gz 2024-08-31T22:55:56.602 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39180.log.gz 2024-08-31T22:55:56.603 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109523.log 2024-08-31T22:55:56.603 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.115015.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.154534.log 2024-08-31T22:55:56.604 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.109523.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93104.log 2024-08-31T22:55:56.604 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.154534.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.154534.log.gz 2024-08-31T22:55:56.605 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161356.log 2024-08-31T22:55:56.605 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.93104.log: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.165415.log.gz 2024-08-31T22:55:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32398.log 2024-08-31T22:55:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161356.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.161356.log.gz 2024-08-31T22:55:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95030.log 2024-08-31T22:55:56.607 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.32398.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93476.log 2024-08-31T22:55:56.607 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28824.log 2024-08-31T22:55:56.608 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.95030.log: /var/log/ceph/cluster2-client.mirror.93476.log: 56.0% -- replaced with /var/log/ceph/cluster2-client.mirror.29420.log.gz 2024-08-31T22:55:56.608 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78122.log 2024-08-31T22:55:56.609 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123608.log 2024-08-31T22:55:56.609 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28824.log: /var/log/ceph/cluster2-client.mirror.78122.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28824.log.gz 2024-08-31T22:55:56.609 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158491.log 2024-08-31T22:55:56.610 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89051.log 2024-08-31T22:55:56.611 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.123608.log: /var/log/ceph/cluster2-client.mirror.158491.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30995.log.gz 2024-08-31T22:55:56.611 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31088.log.gz 2024-08-31T22:55:56.612 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.158491.log.gz 2024-08-31T22:55:56.612 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98208.log 2024-08-31T22:55:56.612 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94561.log.gz 2024-08-31T22:55:56.612 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89051.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114515.log 2024-08-31T22:55:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.98208.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56832.log.gz 2024-08-31T22:55:56.613 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55898.log.gz 2024-08-31T22:55:56.614 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.96150.log.gz 2024-08-31T22:55:56.614 INFO:teuthology.orchestra.run.smithi032.stderr: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.29130.log.gz 2024-08-31T22:55:56.615 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.105938.log.gz 2024-08-31T22:55:56.615 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42736.log 2024-08-31T22:55:56.622 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.114515.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153433.log 2024-08-31T22:55:56.623 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.42736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121133.log 2024-08-31T22:55:56.627 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.153433.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.49989.log.gz 2024-08-31T22:55:56.635 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58983.log 2024-08-31T22:55:56.644 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.121133.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.89051.log.gz 2024-08-31T22:55:56.645 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.78122.log.gz 2024-08-31T22:55:56.646 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32398.log.gz 2024-08-31T22:55:56.647 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70443.log.gz 2024-08-31T22:55:56.648 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.93383.log.gz 2024-08-31T22:55:56.650 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.93476.log.gz 2024-08-31T22:55:56.651 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29437.log 2024-08-31T22:55:56.651 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107434.log.gz 2024-08-31T22:55:56.651 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.111430.log.gz 2024-08-31T22:55:56.652 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58983.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.150847.log.gz 2024-08-31T22:55:56.652 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103791.log 2024-08-31T22:55:56.653 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.29437.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164679.log 2024-08-31T22:55:56.653 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.103791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75165.log 2024-08-31T22:55:56.654 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164679.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.35386.log.gz 2024-08-31T22:55:56.654 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87241.log.gz 2024-08-31T22:55:56.655 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.164679.log.gz 2024-08-31T22:55:56.655 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139407.log 2024-08-31T22:55:56.656 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.75165.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96185.log 2024-08-31T22:55:56.658 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.139407.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81486.log.gz 2024-08-31T22:55:56.658 INFO:teuthology.orchestra.run.smithi032.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.139407.log.gz 2024-08-31T22:55:56.658 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163763.log 2024-08-31T22:55:56.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 has been restored 2024-08-31T22:55:56.659 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96185.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43670.log 2024-08-31T22:55:56.660 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163763.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30588.log.gz 2024-08-31T22:55:56.660 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.163763.log.gz 2024-08-31T22:55:56.661 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111080.log 2024-08-31T22:55:56.661 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.43670.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89237.log 2024-08-31T22:55:56.663 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.111080.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.114267.log.gz 2024-08-31T22:55:56.663 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.114515.log.gz 2024-08-31T22:55:56.663 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.114139.log.gz 2024-08-31T22:55:56.664 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93627.log 2024-08-31T22:55:56.665 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89237.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.114980.log.gz 2024-08-31T22:55:56.665 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115015.log.gz 2024-08-31T22:55:56.665 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83075.log.gz 2024-08-31T22:55:56.665 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132017.log 2024-08-31T22:55:56.666 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.93627.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97026.log.gz 2024-08-31T22:55:56.666 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97084.log.gz 2024-08-31T22:55:56.666 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47315.log 2024-08-31T22:55:56.667 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.132017.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.98208.log.gz 2024-08-31T22:55:56.674 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.90956.log.gz 2024-08-31T22:55:56.674 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65153.log.gz 2024-08-31T22:55:56.679 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61002.log.gz 2024-08-31T22:55:56.683 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71940.log.gz 2024-08-31T22:55:56.683 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.102537.log.gz 2024-08-31T22:55:56.683 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.162978.log.gz 2024-08-31T22:55:56.683 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.75165.log.gz 2024-08-31T22:55:56.683 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72533.log.gz 2024-08-31T22:55:56.684 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.47315.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74777.log.gz 2024-08-31T22:55:56.684 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103610.log.gz 2024-08-31T22:55:56.684 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86683.log 2024-08-31T22:55:56.686 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104198.log.gz 2024-08-31T22:55:56.687 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.110798.log.gz 2024-08-31T22:55:56.687 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.41384.log.gz 2024-08-31T22:55:56.687 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.96185.log.gz 2024-08-31T22:55:56.690 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.113523.log.gz 2024-08-31T22:55:56.695 INFO:teuthology.orchestra.run.smithi032.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.131383.log.gz 2024-08-31T22:55:56.699 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.165763.log 2024-08-31T22:55:56.701 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86683.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161851.log 2024-08-31T22:55:56.702 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.165763.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.165763.log.gz 2024-08-31T22:55:56.703 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152654.log 2024-08-31T22:55:56.703 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161851.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48156.log.gz 2024-08-31T22:55:56.704 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.161851.log.gz 2024-08-31T22:55:56.704 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161896.log 2024-08-31T22:55:56.705 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.152654.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.152654.log.gz 2024-08-31T22:55:56.705 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156763.log 2024-08-31T22:55:56.706 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161896.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.161896.log.gz 2024-08-31T22:55:56.706 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77994.log 2024-08-31T22:55:56.708 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156763.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.156763.log.gz 2024-08-31T22:55:56.709 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73374.log 2024-08-31T22:55:56.709 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77994.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44662.log 2024-08-31T22:55:56.713 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.73374.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41709.log.gz 2024-08-31T22:55:56.716 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.77564.log.gz 2024-08-31T22:55:56.716 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.77750.log.gz 2024-08-31T22:55:56.719 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94844.log 2024-08-31T22:55:56.719 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68296.log.gz 2024-08-31T22:55:56.720 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68389.log.gz 2024-08-31T22:55:56.720 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72219.log 2024-08-31T22:55:56.720 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.44662.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.77994.log.gz 2024-08-31T22:55:56.720 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36134.log.gz 2024-08-31T22:55:56.721 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94844.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156863.log 2024-08-31T22:55:56.721 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.72219.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165032.log 2024-08-31T22:55:56.723 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156863.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.156863.log.gz 2024-08-31T22:55:56.723 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110550.log 2024-08-31T22:55:56.724 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165032.log: 87.9% 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.122161.log.gz 2024-08-31T22:55:56.724 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.165032.log.gz 2024-08-31T22:55:56.724 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.164637.log 2024-08-31T22:55:56.725 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.110550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73188.log 2024-08-31T22:55:56.725 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.164637.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107992.log 2024-08-31T22:55:56.725 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.164637.log.gz/var/log/ceph/cluster2-client.mirror.73188.log: 2024-08-31T22:55:56.725 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164065.log 2024-08-31T22:55:56.726 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107992.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59634.log 2024-08-31T22:55:56.726 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164065.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114736.log 2024-08-31T22:55:56.726 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.59634.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.164065.log.gz 2024-08-31T22:55:56.727 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28109.log 2024-08-31T22:55:56.728 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.114736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158804.log 2024-08-31T22:55:56.728 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28109.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166553.log 2024-08-31T22:55:56.728 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28109.log.gz 2024-08-31T22:55:56.729 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158804.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71975.log 2024-08-31T22:55:56.729 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.166553.log: 0.0% 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.51228.log.gz 2024-08-31T22:55:56.729 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.158804.log.gz 2024-08-31T22:55:56.730 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.166553.log.gz 2024-08-31T22:55:56.730 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41048.log 2024-08-31T22:55:56.730 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94282.log 2024-08-31T22:55:56.730 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.71975.log: /var/log/ceph/cluster2-client.mirror.41048.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51755.log.gz 2024-08-31T22:55:56.731 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.93627.log.gz 2024-08-31T22:55:56.731 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114829.log 2024-08-31T22:55:56.731 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157987.log 2024-08-31T22:55:56.732 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94282.log: /var/log/ceph/cluster2-client.mirror.114829.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.109523.log.gz 88.5% 2024-08-31T22:55:56.732 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.81579.log.gz 2024-08-31T22:55:56.732 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83505.log.gz 2024-08-31T22:55:56.732 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.95030.log.gz 2024-08-31T22:55:56.732 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142396.log 2024-08-31T22:55:56.732 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.98801.log.gz 2024-08-31T22:55:56.732 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157987.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91342.log 2024-08-31T22:55:56.734 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.142396.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.157987.log.gz 2024-08-31T22:55:56.734 INFO:teuthology.orchestra.run.smithi032.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.142396.log.gz 2024-08-31T22:55:56.734 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55212.log 2024-08-31T22:55:56.734 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.91342.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69637.log 2024-08-31T22:55:56.735 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31181.log 2024-08-31T22:55:56.735 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.55212.log: /var/log/ceph/cluster2-client.mirror.69637.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52968.log 2024-08-31T22:55:56.736 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74963.log 2024-08-31T22:55:56.736 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.31181.log: /var/log/ceph/cluster2-client.mirror.52968.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151589.log 2024-08-31T22:55:56.737 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.74963.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80145.log 2024-08-31T22:55:56.737 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45224.log 2024-08-31T22:55:56.737 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.151589.log: /var/log/ceph/cluster2-client.mirror.80145.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145339.log 2024-08-31T22:55:56.738 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.45224.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39459.log 2024-08-31T22:55:56.739 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.145339.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.151589.log.gz 2024-08-31T22:55:56.740 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.154626.log 2024-08-31T22:55:56.740 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39459.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.145339.log.gz 2024-08-31T22:55:56.740 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42143.log 2024-08-31T22:55:56.741 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70292.log 2024-08-31T22:55:56.741 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.154626.log: /var/log/ceph/cluster2-client.mirror.42143.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.154476.log 2024-08-31T22:55:56.742 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.70292.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75940.log 2024-08-31T22:55:56.743 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.154476.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.154476.log.gz 2024-08-31T22:55:56.743 INFO:teuthology.orchestra.run.smithi032.stderr: 91.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63940.log 2024-08-31T22:55:56.743 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.154626.log.gz 2024-08-31T22:55:56.744 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75940.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108930.log 2024-08-31T22:55:56.744 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59789.log 2024-08-31T22:55:56.747 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63940.log: /var/log/ceph/cluster2-client.mirror.108930.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145945.log 2024-08-31T22:55:56.747 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.59789.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130900.log 2024-08-31T22:55:56.747 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88586.log 2024-08-31T22:55:56.749 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.145945.log: /var/log/ceph/cluster2-client.mirror.130900.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.145945.log.gz 2024-08-31T22:55:56.753 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95402.log 2024-08-31T22:55:56.753 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.88586.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92600.log 2024-08-31T22:55:56.754 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.95402.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58235.log 2024-08-31T22:55:56.755 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.92600.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99979.log 2024-08-31T22:55:56.755 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58235.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35851.log 2024-08-31T22:55:56.756 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.99979.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83354.log 2024-08-31T22:55:56.757 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.35851.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29386.log 2024-08-31T22:55:56.757 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.153433.log.gz 2024-08-31T22:55:56.758 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160861.log 2024-08-31T22:55:56.758 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83354.log: /var/log/ceph/cluster2-client.mirror.29386.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.155777.log 2024-08-31T22:55:56.759 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.160861.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.160861.log.gz 2024-08-31T22:55:56.759 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151512.log 2024-08-31T22:55:56.760 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.155777.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72781.log 2024-08-31T22:55:56.761 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.155777.log.gz/var/log/ceph/cluster2-client.mirror.151512.log: 2024-08-31T22:55:56.761 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106217.log 2024-08-31T22:55:56.761 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.151512.log.gz 2024-08-31T22:55:56.763 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.72781.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85594.log 2024-08-31T22:55:56.763 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.106217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40393.log 2024-08-31T22:55:56.764 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.85594.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42457.log 2024-08-31T22:55:56.765 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.40393.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161828.log 2024-08-31T22:55:56.767 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.42457.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112926.log 2024-08-31T22:55:56.767 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161828.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.161828.log.gz 2024-08-31T22:55:56.767 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160906.log 2024-08-31T22:55:56.768 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.112926.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.164604.log 2024-08-31T22:55:56.768 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.160906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146123.log 2024-08-31T22:55:56.769 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.160906.log.gz/var/log/ceph/cluster1-client.mirror.164604.log: 2024-08-31T22:55:56.769 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139057.log 2024-08-31T22:55:56.769 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.164604.log.gz 2024-08-31T22:55:56.769 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.146123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96898.log 2024-08-31T22:55:56.770 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.139057.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.139057.log.gz 2024-08-31T22:55:56.770 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79339.log 2024-08-31T22:55:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96898.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.146123.log.gz 2024-08-31T22:55:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104477.log 2024-08-31T22:55:56.772 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.79339.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.165790.log 2024-08-31T22:55:56.772 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.104477.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90136.log 2024-08-31T22:55:56.773 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.165790.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146689.log 2024-08-31T22:55:56.773 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.165790.log.gz/var/log/ceph/cluster2-client.mirror.90136.log: 2024-08-31T22:55:56.773 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156370.log 2024-08-31T22:55:56.774 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.146689.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49369.log 2024-08-31T22:55:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156370.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.146689.log.gz 2024-08-31T22:55:56.776 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113953.log 2024-08-31T22:55:56.776 INFO:teuthology.orchestra.run.smithi032.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.156370.log.gz 2024-08-31T22:55:56.776 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147851.log 2024-08-31T22:55:56.777 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.113953.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42085.log 2024-08-31T22:55:56.777 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.147851.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146011.log 2024-08-31T22:55:56.777 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.42085.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84009.log 2024-08-31T22:55:56.779 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.146011.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.146011.log.gz 2024-08-31T22:55:56.779 INFO:teuthology.orchestra.run.smithi032.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.147851.log.gz 2024-08-31T22:55:56.779 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50577.log 2024-08-31T22:55:56.779 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.84009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.166152.log 2024-08-31T22:55:56.780 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50577.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157450.log 2024-08-31T22:55:56.780 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.166152.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111934.log 2024-08-31T22:55:56.780 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.166152.log.gz 2024-08-31T22:55:56.780 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157450.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105845.log 2024-08-31T22:55:56.781 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.111934.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.157450.log.gz 2024-08-31T22:55:56.782 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117857.log 2024-08-31T22:55:56.782 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105845.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103285.log 2024-08-31T22:55:56.782 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.117857.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80424.log 2024-08-31T22:55:56.783 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.103285.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142351.log 2024-08-31T22:55:56.783 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.80424.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103702.log 2024-08-31T22:55:56.783 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.142351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140674.log 2024-08-31T22:55:56.784 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.103702.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142351.log.gz 2024-08-31T22:55:56.784 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118728.log 2024-08-31T22:55:56.784 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.140674.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152747.log 2024-08-31T22:55:56.785 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118728.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.140674.log.gz 2024-08-31T22:55:56.786 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40149.log 2024-08-31T22:55:56.786 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.152747.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110050.log 2024-08-31T22:55:56.787 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.40149.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.152747.log.gz 2024-08-31T22:55:56.787 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.79211.log.gz 2024-08-31T22:55:56.787 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.79339.log.gz 2024-08-31T22:55:56.788 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86683.log.gz 2024-08-31T22:55:56.791 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162571.log 2024-08-31T22:55:56.791 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.110050.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143283.log 2024-08-31T22:55:56.792 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162571.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98111.log 2024-08-31T22:55:56.792 INFO:teuthology.orchestra.run.smithi032.stderr: 86.6%/var/log/ceph/cluster2-client.mirror.143283.log: -- replaced with /var/log/ceph/cluster1-client.mirror.162571.log.gz 2024-08-31T22:55:56.793 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.143283.log.gz 2024-08-31T22:55:56.793 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.165139.log 2024-08-31T22:55:56.793 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.98111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68238.log 2024-08-31T22:55:56.794 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.165139.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90558.log 2024-08-31T22:55:56.794 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.68238.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.165139.log.gz 2024-08-31T22:55:56.794 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105318.log 2024-08-31T22:55:56.794 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.90558.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146432.log 2024-08-31T22:55:56.794 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105318.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43015.log 2024-08-31T22:55:56.795 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.146432.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121404.log 2024-08-31T22:55:56.795 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.43015.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.146432.log.gz 2024-08-31T22:55:56.795 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104070.log 2024-08-31T22:55:56.796 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.121404.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75882.log 2024-08-31T22:55:56.796 INFO:teuthology.orchestra.run.smithi032.stderr: 84.6%/var/log/ceph/cluster2-client.mirror.104070.log: -- replaced with /var/log/ceph/cluster2-client.mirror.121404.log.gz 2024-08-31T22:55:56.796 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161243.log 2024-08-31T22:55:56.797 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75882.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76254.log 2024-08-31T22:55:56.798 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161243.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50470.log 2024-08-31T22:55:56.798 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.161243.log.gz/var/log/ceph/cluster2-client.mirror.76254.log: 2024-08-31T22:55:56.798 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138956.log 2024-08-31T22:55:56.798 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50470.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97553.log 2024-08-31T22:55:56.798 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.138956.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65436.log 2024-08-31T22:55:56.799 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.97553.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166319.log 2024-08-31T22:55:56.800 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65436.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71975.log.gz 2024-08-31T22:55:56.801 INFO:teuthology.orchestra.run.smithi032.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.138956.log.gz 2024-08-31T22:55:56.801 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48962.log 2024-08-31T22:55:56.802 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.166319.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.154668.log 2024-08-31T22:55:56.802 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166319.log.gz 2024-08-31T22:55:56.802 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48962.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157350.log 2024-08-31T22:55:56.803 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.154668.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59413.log 2024-08-31T22:55:56.805 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157350.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.157350.log.gz 2024-08-31T22:55:56.807 INFO:teuthology.orchestra.run.smithi032.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.154668.log.gz 2024-08-31T22:55:56.807 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158442.log 2024-08-31T22:55:56.808 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.59413.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87462.log 2024-08-31T22:55:56.808 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158442.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34266.log 2024-08-31T22:55:56.809 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.87462.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.158442.log.gz 2024-08-31T22:55:56.809 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139022.log 2024-08-31T22:55:56.809 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.34266.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135136.log 2024-08-31T22:55:56.810 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.139022.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46195.log 2024-08-31T22:55:56.810 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.135136.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.139022.log.gz 2024-08-31T22:55:56.810 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68947.log 2024-08-31T22:55:56.811 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.46195.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77808.log 2024-08-31T22:55:56.811 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.68947.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.135136.log.gz 2024-08-31T22:55:56.812 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38002.log.gz 2024-08-31T22:55:56.812 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44662.log.gz 2024-08-31T22:55:56.813 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.110550.log.gz 2024-08-31T22:55:56.814 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160793.log 2024-08-31T22:55:56.815 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57766.log 2024-08-31T22:55:56.815 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77808.log: /var/log/ceph/cluster1-client.mirror.160793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160816.log 2024-08-31T22:55:56.815 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.160793.log.gz 2024-08-31T22:55:56.816 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61188.log 2024-08-31T22:55:56.816 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.57766.log: /var/log/ceph/cluster1-client.mirror.160816.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38060.log 2024-08-31T22:55:56.816 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.160816.log.gz 2024-08-31T22:55:56.817 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54557.log 2024-08-31T22:55:56.817 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61188.log: /var/log/ceph/cluster2-client.mirror.38060.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58293.log 2024-08-31T22:55:56.817 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82664.log 2024-08-31T22:55:56.818 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.54557.log: /var/log/ceph/cluster2-client.mirror.58293.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82013.log 2024-08-31T22:55:56.818 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82664.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73002.log 2024-08-31T22:55:56.820 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64653.log.gz 2024-08-31T22:55:56.820 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.73002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99607.log 2024-08-31T22:55:56.820 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82013.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87838.log 2024-08-31T22:55:56.820 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55805.log 2024-08-31T22:55:56.822 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.99607.log: /var/log/ceph/cluster2-client.mirror.87838.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.103877.log.gz 2024-08-31T22:55:56.822 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94282.log.gz 2024-08-31T22:55:56.823 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89613.log 2024-08-31T22:55:56.823 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.55805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159929.log 2024-08-31T22:55:56.824 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38525.log 2024-08-31T22:55:56.824 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89613.log: /var/log/ceph/cluster2-client.mirror.159929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64184.log 2024-08-31T22:55:56.824 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.159929.log.gz 2024-08-31T22:55:56.825 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.38525.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.84625.log.gz 2024-08-31T22:55:56.825 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.119319.log.gz 2024-08-31T22:55:56.825 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81207.log 2024-08-31T22:55:56.825 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64184.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75021.log 2024-08-31T22:55:56.825 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112926.log.gz 2024-08-31T22:55:56.826 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.81207.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80366.log 2024-08-31T22:55:56.826 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54278.log 2024-08-31T22:55:56.827 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75021.log: /var/log/ceph/cluster2-client.mirror.80366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118165.log 2024-08-31T22:55:56.827 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58983.log.gz 2024-08-31T22:55:56.827 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59169.log.gz 2024-08-31T22:55:56.827 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.54278.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.59355.log.gz 2024-08-31T22:55:56.828 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59413.log.gz 2024-08-31T22:55:56.828 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115169.log 2024-08-31T22:55:56.829 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.118165.log: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50271.log.gz 2024-08-31T22:55:56.829 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88268.log 2024-08-31T22:55:56.829 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109674.log 2024-08-31T22:55:56.830 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.115169.log: /var/log/ceph/cluster2-client.mirror.88268.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55154.log 2024-08-31T22:55:56.830 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.164852.log 2024-08-31T22:55:56.831 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.109674.log: /var/log/ceph/cluster2-client.mirror.55154.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.47315.log.gz 2024-08-31T22:55:56.831 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47970.log.gz 2024-08-31T22:55:56.831 INFO:teuthology.orchestra.run.smithi032.stderr: 94.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50340.log.gz 2024-08-31T22:55:56.834 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159858.log 2024-08-31T22:55:56.834 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.164852.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47280.log 2024-08-31T22:55:56.835 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159858.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.89237.log.gz 2024-08-31T22:55:56.835 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.159858.log.gz 2024-08-31T22:55:56.836 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.164852.log.gz 2024-08-31T22:55:56.836 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148924.log 2024-08-31T22:55:56.837 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.89613.log.gz 2024-08-31T22:55:56.837 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.47280.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135091.log 2024-08-31T22:55:56.837 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.148924.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134749.log 2024-08-31T22:55:56.837 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.135091.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.148924.log.gz 2024-08-31T22:55:56.837 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55340.log.gz 2024-08-31T22:55:56.837 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135091.log.gz 2024-08-31T22:55:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108461.log 2024-08-31T22:55:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.134749.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102688.log 2024-08-31T22:55:56.838 INFO:teuthology.orchestra.run.smithi032.stderr: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.134749.log.gz 2024-08-31T22:55:56.839 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.108461.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56797.log 2024-08-31T22:55:56.839 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.102688.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161648.log 2024-08-31T22:55:56.839 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.56797.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60347.log 2024-08-31T22:55:56.840 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161648.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.161648.log.gz 2024-08-31T22:55:56.840 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.77808.log.gz 2024-08-31T22:55:56.840 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58948.log 2024-08-31T22:55:56.841 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60347.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55805.log.gz 2024-08-31T22:55:56.841 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40149.log.gz 2024-08-31T22:55:56.841 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41048.log.gz 2024-08-31T22:55:56.842 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.108461.log.gz 2024-08-31T22:55:56.842 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.108930.log.gz 2024-08-31T22:55:56.842 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.114550.log.gz 2024-08-31T22:55:56.843 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.85594.log.gz 2024-08-31T22:55:56.843 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.114829.log.gz 2024-08-31T22:55:56.843 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.110050.log.gz 2024-08-31T22:55:56.843 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65866.log.gz 2024-08-31T22:55:56.843 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.83354.log.gz 2024-08-31T22:55:56.844 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.95309.log.gz 2024-08-31T22:55:56.844 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.95402.log.gz 2024-08-31T22:55:56.844 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.98111.log.gz 2024-08-31T22:55:56.846 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45224.log.gz 2024-08-31T22:55:56.846 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88268.log.gz 2024-08-31T22:55:56.846 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57766.log.gz 2024-08-31T22:55:56.847 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81393.log 2024-08-31T22:55:56.847 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58948.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54929.log 2024-08-31T22:55:56.848 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.81393.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95216.log 2024-08-31T22:55:56.849 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.54929.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54278.log.gz 2024-08-31T22:55:56.850 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65436.log.gz 2024-08-31T22:55:56.853 INFO:teuthology.orchestra.run.smithi032.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.123884.log.gz 2024-08-31T22:55:56.854 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70723.log 2024-08-31T22:55:56.855 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.95216.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59320.log 2024-08-31T22:55:56.855 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64184.log.gz 2024-08-31T22:55:56.855 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73002.log.gz 2024-08-31T22:55:56.855 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73374.log.gz 2024-08-31T22:55:56.856 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73622.log.gz 2024-08-31T22:55:56.856 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.80145.log.gz 2024-08-31T22:55:56.856 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73560.log.gz 2024-08-31T22:55:56.856 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.70723.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71882.log 2024-08-31T22:55:56.857 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102688.log.gz 2024-08-31T22:55:56.857 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81765.log 2024-08-31T22:55:56.857 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.59320.log: /var/log/ceph/cluster2-client.mirror.71882.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95999.log 2024-08-31T22:55:56.858 INFO:teuthology.orchestra.run.smithi032.stderr: 56.3% -- replaced with /var/log/ceph/cluster2-client.mirror.29437.log.gz 2024-08-31T22:55:56.858 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.81765.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99979.log.gz 2024-08-31T22:55:56.858 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39459.log.gz 2024-08-31T22:55:56.858 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25126.log 2024-08-31T22:55:56.859 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31181.log.gz 2024-08-31T22:55:56.859 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59320.log.gz 2024-08-31T22:55:56.859 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.84009.log.gz 2024-08-31T22:55:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.95999.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28373.log 2024-08-31T22:55:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.25126.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25126.log.gz 2024-08-31T22:55:56.860 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108403.log 2024-08-31T22:55:56.860 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28373.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28373.log.gz 2024-08-31T22:55:56.860 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64219.log 2024-08-31T22:55:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.108403.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106593.log 2024-08-31T22:55:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64219.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.121133.log.gz 2024-08-31T22:55:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150391.log 2024-08-31T22:55:56.862 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.106593.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125944.log 2024-08-31T22:55:56.862 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164761.log 2024-08-31T22:55:56.862 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.150391.log: /var/log/ceph/cluster2-client.mirror.125944.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150391.log.gz 2024-08-31T22:55:56.863 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131139.log 2024-08-31T22:55:56.863 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164761.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.164761.log.gz 2024-08-31T22:55:56.864 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52410.log 2024-08-31T22:55:56.867 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.131139.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65994.log 2024-08-31T22:55:56.867 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.52410.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83974.log 2024-08-31T22:55:56.868 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164394.log 2024-08-31T22:55:56.868 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65994.log: /var/log/ceph/cluster2-client.mirror.83974.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35537.log 2024-08-31T22:55:56.868 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166605.log 2024-08-31T22:55:56.869 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164394.log: /var/log/ceph/cluster2-client.mirror.35537.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114325.log 2024-08-31T22:55:56.869 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.166605.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166605.log.gz 2024-08-31T22:55:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65339.log 2024-08-31T22:55:56.871 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.114325.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.164394.log.gz 2024-08-31T22:55:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75133.log 2024-08-31T22:55:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65339.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75403.log 2024-08-31T22:55:56.873 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71471.log 2024-08-31T22:55:56.873 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75133.log: /var/log/ceph/cluster2-client.mirror.75403.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154130.log 2024-08-31T22:55:56.874 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135326.log 2024-08-31T22:55:56.874 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.71471.log: /var/log/ceph/cluster1-client.mirror.154130.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.154130.log.gz 2024-08-31T22:55:56.875 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159381.log 2024-08-31T22:55:56.878 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.135326.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.132017.log.gz 2024-08-31T22:55:56.878 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31743.log 2024-08-31T22:55:56.879 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92321.log 2024-08-31T22:55:56.879 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159381.log: 95.1% -- replaced with /var/log/ceph/cluster2-client.mirror.130900.log.gz 2024-08-31T22:55:56.880 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.31743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160838.log 2024-08-31T22:55:56.881 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.92321.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76161.log 2024-08-31T22:55:56.881 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.159381.log.gz 2024-08-31T22:55:56.881 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.160838.log: 96.5% -- replaced with /var/log/ceph/cluster2-client.mirror.135326.log.gz 2024-08-31T22:55:56.881 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.160838.log.gz 2024-08-31T22:55:56.881 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74366.log 2024-08-31T22:55:56.881 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.76161.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159796.log 2024-08-31T22:55:56.883 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/cluster2-client.mirror.74366.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159534.log 2024-08-31T22:55:56.883 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159796.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81451.log 2024-08-31T22:55:56.883 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159534.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159605.log 2024-08-31T22:55:56.883 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3%/var/log/ceph/cluster2-client.mirror.81451.log: -- replaced with /var/log/ceph/cluster2-client.mirror.159796.log.gz 2024-08-31T22:55:56.883 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.159534.log.gz 2024-08-31T22:55:56.883 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165389.log 2024-08-31T22:55:56.883 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159119.log 2024-08-31T22:55:56.886 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159605.log: /var/log/ceph/cluster2-client.mirror.165389.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105415.log 2024-08-31T22:55:56.886 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.159605.log.gz 2024-08-31T22:55:56.886 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159119.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47001.log 2024-08-31T22:55:56.886 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.165389.log.gz 2024-08-31T22:55:56.886 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105415.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73808.log 2024-08-31T22:55:56.886 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.159119.log.gz 2024-08-31T22:55:56.886 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87183.log 2024-08-31T22:55:56.886 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.47001.log: /var/log/ceph/cluster2-client.mirror.73808.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124869.log 2024-08-31T22:55:56.886 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.87183.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63250.log 2024-08-31T22:55:56.886 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.124869.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34173.log 2024-08-31T22:55:56.886 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63250.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164044.log 2024-08-31T22:55:56.888 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.34173.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42050.log 2024-08-31T22:55:56.888 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164044.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45538.log 2024-08-31T22:55:56.888 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.42050.log: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78742.log 2024-08-31T22:55:56.888 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.164044.log.gz 2024-08-31T22:55:56.888 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79021.log 2024-08-31T22:55:56.888 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.45538.log: /var/log/ceph/cluster2-client.mirror.78742.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113767.log 2024-08-31T22:55:56.889 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.79021.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95437.log 2024-08-31T22:55:56.890 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.113767.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151855.log 2024-08-31T22:55:56.890 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.95437.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85187.log 2024-08-31T22:55:56.890 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.151855.log: /var/log/ceph/cluster2-client.mirror.85187.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47280.log.gz 2024-08-31T22:55:56.890 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.151855.log.gz 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.58235.log.gz 2024-08-31T22:55:56.890 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:56.891 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164353.log 2024-08-31T22:55:56.891 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49090.log 2024-08-31T22:55:56.891 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164353.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64595.log 2024-08-31T22:55:56.892 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49090.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.164353.log.gz 2024-08-31T22:55:56.892 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40711.log 2024-08-31T22:55:56.892 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64595.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35758.log 2024-08-31T22:55:56.892 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.40711.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40300.log 2024-08-31T22:55:56.892 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.35758.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149601.log 2024-08-31T22:55:56.893 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97212.log 2024-08-31T22:55:56.895 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.149601.log: /var/log/ceph/cluster2-client.mirror.40300.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34324.log 2024-08-31T22:55:56.895 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157401.log 2024-08-31T22:55:56.900 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.97212.log: /var/log/ceph/cluster2-client.mirror.34324.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52968.log.gz 2024-08-31T22:55:56.901 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31743.log.gz 2024-08-31T22:55:56.901 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59917.log 2024-08-31T22:55:56.902 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157401.log: 96.8% -- replaced with /var/log/ceph/cluster2-client.mirror.149601.log.gz 2024-08-31T22:55:56.902 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35758.log.gz 2024-08-31T22:55:56.902 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.157401.log.gz 2024-08-31T22:55:56.902 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35851.log.gz 2024-08-31T22:55:56.902 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47745.log 2024-08-31T22:55:56.902 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.59917.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88644.log 2024-08-31T22:55:56.903 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.47745.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87148.log 2024-08-31T22:55:56.904 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.88644.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72126.log 2024-08-31T22:55:56.904 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.87148.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102165.log 2024-08-31T22:55:56.904 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.72126.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91126.log 2024-08-31T22:55:56.904 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.102165.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56460.log 2024-08-31T22:55:56.905 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.91126.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136579.log 2024-08-31T22:55:56.905 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.56460.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39087.log 2024-08-31T22:55:56.905 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.136579.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69695.log 2024-08-31T22:55:56.905 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.136579.log.gz 2024-08-31T22:55:56.905 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39087.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97553.log.gz 2024-08-31T22:55:56.907 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.108682.log.gz 2024-08-31T22:55:56.907 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108089.log.gz 2024-08-31T22:55:56.907 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91274.log 2024-08-31T22:55:56.908 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.69695.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87652.log 2024-08-31T22:55:56.908 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.91274.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35572.log 2024-08-31T22:55:56.908 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.87652.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30623.log 2024-08-31T22:55:56.909 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.35572.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119189.log 2024-08-31T22:55:56.910 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.30623.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119126.log 2024-08-31T22:55:56.910 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119189.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46439.log 2024-08-31T22:55:56.910 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119126.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151890.log 2024-08-31T22:55:56.910 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.46439.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40242.log 2024-08-31T22:55:56.911 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.151890.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.151890.log.gz 2024-08-31T22:55:56.911 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142376.log 2024-08-31T22:55:56.912 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.40242.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165195.log 2024-08-31T22:55:56.912 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.142376.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58142.log 2024-08-31T22:55:56.913 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165195.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.142376.log.gz 2024-08-31T22:55:56.916 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47408.log 2024-08-31T22:55:56.916 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58142.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151183.log 2024-08-31T22:55:56.916 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108368.log 2024-08-31T22:55:56.917 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.47408.log: /var/log/ceph/cluster2-client.mirror.151183.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114325.log.gz 2024-08-31T22:55:56.918 INFO:teuthology.orchestra.run.smithi032.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.151183.log.gz 2024-08-31T22:55:56.919 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122707.log 2024-08-31T22:55:56.919 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.108368.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73750.log 2024-08-31T22:55:56.923 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.122707.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75940.log.gz 2024-08-31T22:55:56.924 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.85187.log.gz 2024-08-31T22:55:56.929 INFO:teuthology.orchestra.run.smithi032.stderr: 94.1% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-08-31T22:55:56.930 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.92135.log.gz 2024-08-31T22:55:56.931 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.92600.log.gz 2024-08-31T22:55:56.931 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158462.log 2024-08-31T22:55:56.933 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.73750.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93104.log.gz 2024-08-31T22:55:56.933 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.94844.log.gz 2024-08-31T22:55:56.933 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112585.log 2024-08-31T22:55:56.933 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158462.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.95216.log.gz 2024-08-31T22:55:56.935 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39029.log.gz 2024-08-31T22:55:56.935 INFO:teuthology.orchestra.run.smithi032.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.158462.log.gz 2024-08-31T22:55:56.936 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45631.log 2024-08-31T22:55:56.936 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.112585.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43670.log.gz 2024-08-31T22:55:56.937 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63378.log 2024-08-31T22:55:56.938 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.45631.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46009.log.gz 2024-08-31T22:55:56.938 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46067.log.gz 2024-08-31T22:55:56.938 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45881.log.gz 2024-08-31T22:55:56.939 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123585.log 2024-08-31T22:55:56.939 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.96898.log.gz 2024-08-31T22:55:56.940 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63378.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43015.log.gz 2024-08-31T22:55:56.941 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.81765.log.gz 2024-08-31T22:55:56.942 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.90136.log.gz 2024-08-31T22:55:56.943 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75237.log 2024-08-31T22:55:56.943 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.123585.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63940.log.gz 2024-08-31T22:55:56.943 INFO:teuthology.orchestra.run.smithi032.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123585.log.gz 2024-08-31T22:55:56.944 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61188.log.gz 2024-08-31T22:55:56.944 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70723.log.gz 2024-08-31T22:55:56.945 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.111934.log.gz 2024-08-31T22:55:56.945 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73750.log.gz 2024-08-31T22:55:56.945 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64688.log.gz 2024-08-31T22:55:56.946 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74963.log.gz 2024-08-31T22:55:56.946 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164877.log 2024-08-31T22:55:56.946 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75237.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.106217.log.gz 2024-08-31T22:55:56.948 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38060.log.gz 2024-08-31T22:55:56.948 INFO:teuthology.orchestra.run.smithi032.stderr: 64.7% -- replaced with /var/log/ceph/cluster2-client.mirror.29386.log.gz 2024-08-31T22:55:56.949 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.95999.log.gz 2024-08-31T22:55:56.949 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39087.log.gz 2024-08-31T22:55:56.949 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104477.log.gz 2024-08-31T22:55:56.949 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105415.log.gz 2024-08-31T22:55:56.949 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59917.log.gz 2024-08-31T22:55:56.950 INFO:teuthology.orchestra.run.smithi032.stderr: 96.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115259.log.gz 2024-08-31T22:55:56.950 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86435.log 2024-08-31T22:55:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164877.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.164877.log.gz 2024-08-31T22:55:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56987.log 2024-08-31T22:55:56.953 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86435.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50251.log 2024-08-31T22:55:56.953 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.56987.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119542.log 2024-08-31T22:55:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50251.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77246.log 2024-08-31T22:55:56.955 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119542.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63006.log 2024-08-31T22:55:56.955 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77246.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67920.log 2024-08-31T22:55:56.956 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63006.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29577.log 2024-08-31T22:55:56.957 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67920.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60944.log 2024-08-31T22:55:56.959 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.29577.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76595.log 2024-08-31T22:55:56.959 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60944.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64746.log 2024-08-31T22:55:56.960 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.76595.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80048.log 2024-08-31T22:55:56.961 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64746.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87369.log 2024-08-31T22:55:56.961 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.80048.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.117857.log.gz 2024-08-31T22:55:56.961 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117915.log.gz 2024-08-31T22:55:56.962 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47408.log.gz 2024-08-31T22:55:56.963 INFO:teuthology.orchestra.run.smithi032.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.123608.log.gz 2024-08-31T22:55:56.964 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163804.log 2024-08-31T22:55:56.964 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82850.log 2024-08-31T22:55:56.964 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.87369.log: /var/log/ceph/cluster2-client.mirror.163804.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135437.log 2024-08-31T22:55:56.965 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47745.log.gz 2024-08-31T22:55:56.965 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146709.log 2024-08-31T22:55:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.135437.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49090.log.gz 2024-08-31T22:55:56.966 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.135437.log.gz 2024-08-31T22:55:56.967 INFO:teuthology.orchestra.run.smithi032.stderr: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.163804.log.gz 2024-08-31T22:55:56.967 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46850.log 2024-08-31T22:55:56.969 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.146709.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.146709.log.gz 2024-08-31T22:55:56.969 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42085.log.gz 2024-08-31T22:55:56.970 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34917.log 2024-08-31T22:55:56.970 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46625.log 2024-08-31T22:55:56.971 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.46850.log: /var/log/ceph/cluster2-client.mirror.34917.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60440.log 2024-08-31T22:55:56.971 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.55212.log.gz 2024-08-31T22:55:56.971 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.77246.log.gz/var/log/ceph/cluster2-client.mirror.46625.log: 2024-08-31T22:55:56.971 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82141.log 2024-08-31T22:55:56.971 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.82385.log.gz 2024-08-31T22:55:56.972 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60440.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40393.log.gz 2024-08-31T22:55:56.975 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101603.log 2024-08-31T22:55:56.975 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83726.log 2024-08-31T22:55:56.976 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101603.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.108403.log.gz 2024-08-31T22:55:56.977 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28142.log 2024-08-31T22:55:56.977 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83726.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68145.log 2024-08-31T22:55:56.978 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90264.log 2024-08-31T22:55:56.978 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28142.log: /var/log/ceph/cluster2-client.mirror.68145.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28142.log.gz 2024-08-31T22:55:56.979 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124143.log 2024-08-31T22:55:56.979 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.90264.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116617.log 2024-08-31T22:55:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92356.log 2024-08-31T22:55:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.124143.log: /var/log/ceph/cluster2-client.mirror.116617.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.124143.log.gz 2024-08-31T22:55:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31894.log 2024-08-31T22:55:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.92356.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120962.log 2024-08-31T22:55:56.982 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.31894.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30681.log 2024-08-31T22:55:56.982 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.81207.log.gz 2024-08-31T22:55:56.983 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.120962.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.113953.log.gz 2024-08-31T22:55:56.983 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42457.log.gz 2024-08-31T22:55:56.983 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86435.log.gz 2024-08-31T22:55:56.983 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.87369.log.gz 2024-08-31T22:55:56.984 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103814.log 2024-08-31T22:55:56.984 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.30681.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68575.log 2024-08-31T22:55:56.984 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59731.log 2024-08-31T22:55:56.984 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.103814.log: /var/log/ceph/cluster2-client.mirror.68575.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91608.log 2024-08-31T22:55:56.985 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141801.log 2024-08-31T22:55:56.985 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.59731.log: /var/log/ceph/cluster2-client.mirror.91608.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51286.log 2024-08-31T22:55:56.986 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.141801.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54088.log 2024-08-31T22:55:56.986 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141801.log.gz 2024-08-31T22:55:56.987 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137693.log 2024-08-31T22:55:56.987 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.54088.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32956.log 2024-08-31T22:55:56.987 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102223.log 2024-08-31T22:55:56.989 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.137693.log: /var/log/ceph/cluster2-client.mirror.32956.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89706.log 2024-08-31T22:55:56.990 INFO:teuthology.orchestra.run.smithi032.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.137693.log.gz 2024-08-31T22:55:56.990 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.102223.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156814.log 2024-08-31T22:55:56.990 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89706.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41257.log 2024-08-31T22:55:56.991 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.164930.log 2024-08-31T22:55:56.991 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156814.log: /var/log/ceph/cluster1-client.mirror.41257.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72161.log 2024-08-31T22:55:56.991 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.156814.log.gz 2024-08-31T22:55:56.992 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.164930.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148503.log 2024-08-31T22:55:56.992 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.164930.log.gz 2024-08-31T22:55:56.992 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.72161.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105473.log 2024-08-31T22:55:56.993 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.148503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146103.log 2024-08-31T22:55:56.993 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.148503.log.gz 2024-08-31T22:55:56.993 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105473.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103343.log 2024-08-31T22:55:56.994 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.82850.log.gz 2024-08-31T22:55:56.994 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76440.log 2024-08-31T22:55:56.995 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.146103.log: /var/log/ceph/cluster2-client.mirror.103343.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63533.log 2024-08-31T22:55:56.996 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.76440.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127450.log 2024-08-31T22:55:56.996 INFO:teuthology.orchestra.run.smithi032.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.146103.log.gz 2024-08-31T22:55:56.996 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161693.log 2024-08-31T22:55:56.997 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63533.log: /var/log/ceph/cluster2-client.mirror.127450.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158633.log 2024-08-31T22:55:56.998 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161693.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.161693.log.gz 2024-08-31T22:55:56.998 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140079.log 2024-08-31T22:55:57.000 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158633.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.158633.log.gz 2024-08-31T22:55:57.001 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143382.log 2024-08-31T22:55:57.002 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.140079.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.140079.log.gz 2024-08-31T22:55:57.002 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62533.log 2024-08-31T22:55:57.006 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.143382.log: 95.4% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63006.log.gz 2024-08-31T22:55:57.006 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.143382.log.gz 2024-08-31T22:55:57.006 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56987.log.gz 2024-08-31T22:55:57.007 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63285.log 2024-08-31T22:55:57.007 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62533.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42736.log.gz 2024-08-31T22:55:57.007 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63378.log.gz 2024-08-31T22:55:57.008 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107562.log 2024-08-31T22:55:57.009 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63285.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63533.log.gz 2024-08-31T22:55:57.009 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.79021.log.gz 2024-08-31T22:55:57.009 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.88586.log.gz 2024-08-31T22:55:57.009 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54557.log.gz 2024-08-31T22:55:57.009 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54929.log.gz 2024-08-31T22:55:57.009 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60347.log.gz 2024-08-31T22:55:57.009 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.60440.log.gz 2024-08-31T22:55:57.010 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64219.log.gz 2024-08-31T22:55:57.010 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.62533.log.gz 2024-08-31T22:55:57.010 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99828.log 2024-08-31T22:55:57.011 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.80424.log.gz 2024-08-31T22:55:57.011 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107562.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71882.log.gz 2024-08-31T22:55:57.011 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72781.log.gz 2024-08-31T22:55:57.011 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74366.log.gz 2024-08-31T22:55:57.012 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125022.log 2024-08-31T22:55:57.012 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.99828.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100762.log 2024-08-31T22:55:57.012 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75133.log.gz 2024-08-31T22:55:57.012 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.80048.log.gz 2024-08-31T22:55:57.012 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101603.log.gz 2024-08-31T22:55:57.013 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.125022.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103343.log.gz 2024-08-31T22:55:57.013 INFO:teuthology.orchestra.run.smithi032.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125022.log.gz 2024-08-31T22:55:57.014 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106124.log 2024-08-31T22:55:57.014 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.100762.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116055.log 2024-08-31T22:55:57.015 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.106124.log: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.119542.log.gz 2024-08-31T22:55:57.015 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89295.log 2024-08-31T22:55:57.017 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.116055.log: 87.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50577.log.gz 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58948.log.gz 2024-08-31T22:55:57.017 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105845.log.gz 2024-08-31T22:55:57.017 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.83974.log.gz 2024-08-31T22:55:57.017 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:57.018 INFO:teuthology.orchestra.run.smithi032.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.131139.log.gz 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106124.log.gz 2024-08-31T22:55:57.018 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:57.018 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146975.log 2024-08-31T22:55:57.019 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46850.log.gz 2024-08-31T22:55:57.019 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89295.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80238.log 2024-08-31T22:55:57.020 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.146975.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.146975.log.gz 2024-08-31T22:55:57.020 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77095.log 2024-08-31T22:55:57.020 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.80238.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89485.log 2024-08-31T22:55:57.023 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77095.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58293.log.gz 2024-08-31T22:55:57.025 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82141.log.gz 2024-08-31T22:55:57.026 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70292.log.gz 2024-08-31T22:55:57.026 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31894.log.gz 2024-08-31T22:55:57.026 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.47001.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.35537.log.gz 2024-08-31T22:55:57.027 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:57.027 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32956.log.gz 2024-08-31T22:55:57.027 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40711.log.gz 2024-08-31T22:55:57.031 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.109674.log.gz 2024-08-31T22:55:57.031 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76161.log.gz 2024-08-31T22:55:57.031 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.81393.log.gz 2024-08-31T22:55:57.032 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.87148.log.gz 2024-08-31T22:55:57.032 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.87462.log.gz 2024-08-31T22:55:57.033 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87838.log.gz 2024-08-31T22:55:57.033 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.92321.log.gz 2024-08-31T22:55:57.034 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107899.log 2024-08-31T22:55:57.035 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89485.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158704.log 2024-08-31T22:55:57.036 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107899.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42143.log.gz 2024-08-31T22:55:57.047 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95685.log 2024-08-31T22:55:57.049 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158704.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.158704.log.gz 2024-08-31T22:55:57.049 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68889.log 2024-08-31T22:55:57.050 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.95685.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164972.log 2024-08-31T22:55:57.050 INFO:teuthology.orchestra.run.smithi032.stderr: 95.2% -- replaced with /var/log/ceph/cluster2-client.mirror.165195.log.gz 2024-08-31T22:55:57.051 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.68889.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50226.log 2024-08-31T22:55:57.051 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82478.log 2024-08-31T22:55:57.052 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3%/var/log/ceph/cluster1-client.mirror.50226.log: -- replaced with /var/log/ceph/cluster2-client.mirror.164972.log.gz 2024-08-31T22:55:57.052 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46660.log 2024-08-31T22:55:57.052 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82478.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83691.log 2024-08-31T22:55:57.053 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.46660.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28083.log 2024-08-31T22:55:57.053 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130458.log 2024-08-31T22:55:57.054 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83691.log: /var/log/ceph/cluster2-client.admin.28083.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28083.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92228.log 2024-08-31T22:55:57.054 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:57.054 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160571.log 2024-08-31T22:55:57.055 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.130458.log: /var/log/ceph/cluster2-client.mirror.92228.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.130458.log.gz 2024-08-31T22:55:57.058 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154987.log 2024-08-31T22:55:57.059 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97274.log 2024-08-31T22:55:57.059 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.160571.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.160571.log.gz 2024-08-31T22:55:57.060 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.154987.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62658.log 2024-08-31T22:55:57.060 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.97274.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.154987.log.gz 2024-08-31T22:55:57.061 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.62658.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53003.log 2024-08-31T22:55:57.061 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158613.log 2024-08-31T22:55:57.062 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114081.log 2024-08-31T22:55:57.062 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53003.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118591.log 2024-08-31T22:55:57.063 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158613.log: /var/log/ceph/cluster2-client.mirror.114081.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.158613.log.gz 2024-08-31T22:55:57.063 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109957.log 2024-08-31T22:55:57.063 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85222.log 2024-08-31T22:55:57.064 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118591.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63661.log 2024-08-31T22:55:57.064 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.109957.log: /var/log/ceph/cluster2-client.mirror.85222.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101103.log 2024-08-31T22:55:57.065 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107934.log 2024-08-31T22:55:57.066 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161761.log 2024-08-31T22:55:57.066 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95158.log 2024-08-31T22:55:57.067 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107934.log: /var/log/ceph/cluster1-client.mirror.161761.log: 86.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99047.log 2024-08-31T22:55:57.067 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.161761.log.gz 2024-08-31T22:55:57.068 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.95158.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162796.log 2024-08-31T22:55:57.068 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31239.log 2024-08-31T22:55:57.068 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.99047.log: /var/log/ceph/cluster1-client.mirror.162796.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.162796.log.gz 2024-08-31T22:55:57.069 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103645.log 2024-08-31T22:55:57.069 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154601.log 2024-08-31T22:55:57.069 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.31239.log: /var/log/ceph/cluster2-client.mirror.103645.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40676.log 2024-08-31T22:55:57.070 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100634.log 2024-08-31T22:55:57.071 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.154601.log: /var/log/ceph/cluster2-client.mirror.40676.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154601.log.gz 2024-08-31T22:55:57.071 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156441.log 2024-08-31T22:55:57.071 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137822.log 2024-08-31T22:55:57.072 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.100634.log: /var/log/ceph/cluster2-client.mirror.156441.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131360.log 2024-08-31T22:55:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.137822.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.137822.log.gz 2024-08-31T22:55:57.075 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59076.log 2024-08-31T22:55:57.075 INFO:teuthology.orchestra.run.smithi032.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.156441.log.gz 2024-08-31T22:55:57.075 INFO:teuthology.orchestra.run.smithi032.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124166.log.gz 2024-08-31T22:55:57.075 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.131360.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50159.log 2024-08-31T22:55:57.075 INFO:teuthology.orchestra.run.smithi032.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131360.log.gz/var/log/ceph/cluster2-client.mirror.59076.log: 2024-08-31T22:55:57.075 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149787.log 2024-08-31T22:55:57.076 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31929.log 2024-08-31T22:55:57.076 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50159.log: /var/log/ceph/cluster2-client.mirror.149787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.163918.log 2024-08-31T22:55:57.076 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63250.log.gz 2024-08-31T22:55:57.076 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119497.log 2024-08-31T22:55:57.076 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.31929.log: /var/log/ceph/cluster1-client.mirror.163918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46381.log 2024-08-31T22:55:57.077 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.163918.log.gz 2024-08-31T22:55:57.077 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119497.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95499.log 2024-08-31T22:55:57.077 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.46381.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.91126.log.gz 2024-08-31T22:55:57.077 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65994.log.gz 2024-08-31T22:55:57.078 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46102.log 2024-08-31T22:55:57.078 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68889.log.gz 2024-08-31T22:55:57.078 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68947.log.gz 2024-08-31T22:55:57.079 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.95499.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104539.log 2024-08-31T22:55:57.079 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.46102.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119604.log 2024-08-31T22:55:57.080 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117660.log 2024-08-31T22:55:57.080 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.104539.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102630.log 2024-08-31T22:55:57.081 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119604.log: /var/log/ceph/cluster1-client.mirror.117660.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72126.log.gz 2024-08-31T22:55:57.081 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75021.log.gz 2024-08-31T22:55:57.081 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53751.log 2024-08-31T22:55:57.082 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155458.log 2024-08-31T22:55:57.082 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.102630.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80610.log 2024-08-31T22:55:57.083 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53751.log: /var/log/ceph/cluster1-client.mirror.155458.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46908.log 2024-08-31T22:55:57.083 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.155458.log.gz 2024-08-31T22:55:57.083 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.80610.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119080.log 2024-08-31T22:55:57.084 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.46908.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57673.log 2024-08-31T22:55:57.084 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138879.log 2024-08-31T22:55:57.084 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119080.log: /var/log/ceph/cluster2-client.mirror.57673.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140191.log 2024-08-31T22:55:57.085 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.155915.log 2024-08-31T22:55:57.085 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.138879.log: /var/log/ceph/cluster1-client.mirror.140191.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.138879.log.gz 2024-08-31T22:55:57.087 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.124869.log.gz 2024-08-31T22:55:57.087 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102165.log.gz 2024-08-31T22:55:57.087 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103285.log.gz 2024-08-31T22:55:57.087 INFO:teuthology.orchestra.run.smithi032.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.140191.log.gz 2024-08-31T22:55:57.090 INFO:teuthology.orchestra.run.smithi032.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119189.log.gz 2024-08-31T22:55:57.091 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41302.log 2024-08-31T22:55:57.092 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.89295.log.gz 2024-08-31T22:55:57.092 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85001.log 2024-08-31T22:55:57.092 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.155915.log: /var/log/ceph/cluster2-client.mirror.41302.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94096.log 2024-08-31T22:55:57.093 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.85001.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104163.log 2024-08-31T22:55:57.093 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2%/var/log/ceph/cluster2-client.mirror.94096.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28012.log 2024-08-31T22:55:57.093 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.155915.log.gz 2024-08-31T22:55:57.093 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.104163.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68331.log 2024-08-31T22:55:57.093 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28012.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68668.log 2024-08-31T22:55:57.093 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28012.log.gz 2024-08-31T22:55:57.095 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.68331.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104725.log 2024-08-31T22:55:57.095 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28899.log 2024-08-31T22:55:57.095 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.68668.log: /var/log/ceph/cluster2-client.mirror.104725.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118521.log 2024-08-31T22:55:57.095 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28899.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142222.log 2024-08-31T22:55:57.095 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.118521.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.105318.log.gz 2024-08-31T22:55:57.095 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28899.log.gz 2024-08-31T22:55:57.096 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88303.log 2024-08-31T22:55:57.097 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.142222.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65060.log 2024-08-31T22:55:57.097 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.88303.log: -- replaced with /var/log/ceph/cluster2-client.mirror.142222.log.gz 2024-08-31T22:55:57.097 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43484.log 2024-08-31T22:55:57.098 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65060.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136930.log 2024-08-31T22:55:57.098 INFO:teuthology.orchestra.run.smithi032.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-client.mirror.29645.log.gz 2024-08-31T22:55:57.098 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.43484.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32173.log 2024-08-31T22:55:57.099 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38095.log 2024-08-31T22:55:57.099 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.136930.log: /var/log/ceph/cluster2-client.mirror.32173.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53844.log 2024-08-31T22:55:57.099 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.136930.log.gz 2024-08-31T22:55:57.100 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37312.log 2024-08-31T22:55:57.100 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.38095.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46625.log.gz/var/log/ceph/cluster2-client.mirror.53844.log: 2024-08-31T22:55:57.100 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.49369.log.gz 2024-08-31T22:55:57.100 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65025.log 2024-08-31T22:55:57.100 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46660.log.gz 2024-08-31T22:55:57.100 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.37312.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57173.log 2024-08-31T22:55:57.101 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65025.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139813.log 2024-08-31T22:55:57.101 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41176.log 2024-08-31T22:55:57.101 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.57173.log: /var/log/ceph/cluster1-client.mirror.139813.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71657.log 2024-08-31T22:55:57.102 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.41176.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105566.log 2024-08-31T22:55:57.103 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.71657.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164262.log 2024-08-31T22:55:57.103 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105566.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118611.log 2024-08-31T22:55:57.104 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32080.log 2024-08-31T22:55:57.104 INFO:teuthology.orchestra.run.smithi032.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.139813.log.gzgzip 2024-08-31T22:55:57.104 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118008.log 2024-08-31T22:55:57.104 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.118611.log: /var/log/ceph/cluster2-client.mirror.32080.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86904.log 2024-08-31T22:55:57.104 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118008.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138911.log 2024-08-31T22:55:57.105 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29096.log 2024-08-31T22:55:57.105 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.138911.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158349.log 2024-08-31T22:55:57.105 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.29096.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138911.log.gz 2024-08-31T22:55:57.105 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151225.log 2024-08-31T22:55:57.106 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158349.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83447.log 2024-08-31T22:55:57.106 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.151225.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51286.log.gz 2024-08-31T22:55:57.106 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.158349.log.gz 2024-08-31T22:55:57.106 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58142.log.gz 2024-08-31T22:55:57.107 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74401.log 2024-08-31T22:55:57.107 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83447.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37816.log 2024-08-31T22:55:57.108 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.74401.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113581.log 2024-08-31T22:55:57.108 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.37816.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163487.log 2024-08-31T22:55:57.109 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.113581.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48400.log 2024-08-31T22:55:57.111 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163487.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42050.log.gz 2024-08-31T22:55:57.111 INFO:teuthology.orchestra.run.smithi032.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.163487.log.gz 2024-08-31T22:55:57.111 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129671.log 2024-08-31T22:55:57.112 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70948.log 2024-08-31T22:55:57.117 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48400.log: /var/log/ceph/cluster2-client.mirror.129671.log: 87.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40242.log.gz 2024-08-31T22:55:57.117 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40676.log.gz 2024-08-31T22:55:57.119 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118341.log 2024-08-31T22:55:57.119 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.70948.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40490.log.gz 2024-08-31T22:55:57.120 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.155391.log 2024-08-31T22:55:57.120 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.118341.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112833.log 2024-08-31T22:55:57.121 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.155391.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.155391.log.gz 2024-08-31T22:55:57.122 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49896.log 2024-08-31T22:55:57.122 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.112833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109581.log 2024-08-31T22:55:57.123 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49896.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44848.log 2024-08-31T22:55:57.123 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32173.log.gz 2024-08-31T22:55:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.109581.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89388.log 2024-08-31T22:55:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.44848.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28799.log 2024-08-31T22:55:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89388.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37626.log 2024-08-31T22:55:57.126 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28799.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28799.log.gz 2024-08-31T22:55:57.126 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135402.log 2024-08-31T22:55:57.127 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.37626.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86121.log 2024-08-31T22:55:57.128 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.135402.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.135402.log.gz 2024-08-31T22:55:57.128 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116338.log 2024-08-31T22:55:57.129 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86121.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63905.log 2024-08-31T22:55:57.129 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.116338.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159878.log 2024-08-31T22:55:57.130 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63905.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35723.log 2024-08-31T22:55:57.134 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159878.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.159878.log.gz 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.85222.log.gz 2024-08-31T22:55:57.134 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.87183.log.gz 2024-08-31T22:55:57.134 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.92228.log.gz 2024-08-31T22:55:57.134 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:57.134 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87652.log.gz 2024-08-31T22:55:57.135 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28299.log 2024-08-31T22:55:57.135 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.35723.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49803.log 2024-08-31T22:55:57.136 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28299.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28299.log.gz 2024-08-31T22:55:57.136 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50117.log 2024-08-31T22:55:57.137 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49803.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60196.log 2024-08-31T22:55:57.137 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50117.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91319.log 2024-08-31T22:55:57.139 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60196.log: 95.2% -- replaced with /var/log/ceph/cluster2-client.mirror.164262.log.gz 2024-08-31T22:55:57.140 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62971.log 2024-08-31T22:55:57.141 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.91319.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152241.log 2024-08-31T22:55:57.141 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62971.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67304.log 2024-08-31T22:55:57.141 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.152241.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114046.log 2024-08-31T22:55:57.142 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67304.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.152241.log.gz 2024-08-31T22:55:57.142 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.163052.log 2024-08-31T22:55:57.142 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.114046.log: /var/log/ceph/cluster2-client.mirror.115776.log 2024-08-31T22:55:57.143 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57545.log 2024-08-31T22:55:57.143 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.163052.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.95437.log.gz 2024-08-31T22:55:57.143 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.115776.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.163052.log.gz 2024-08-31T22:55:57.143 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94503.log 2024-08-31T22:55:57.143 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.57545.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82292.log 2024-08-31T22:55:57.145 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46567.log 2024-08-31T22:55:57.145 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.95499.log.gz 2024-08-31T22:55:57.145 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82292.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161738.log 2024-08-31T22:55:57.145 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.46567.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41523.log 2024-08-31T22:55:57.146 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161738.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.117660.log.gz 2024-08-31T22:55:57.146 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.95685.log.gz 2024-08-31T22:55:57.146 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.161738.log.gz 2024-08-31T22:55:57.146 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.97274.log.gz 2024-08-31T22:55:57.146 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87427.log 2024-08-31T22:55:57.146 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.41523.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58514.log 2024-08-31T22:55:57.146 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.87427.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67114.log 2024-08-31T22:55:57.147 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144595.log 2024-08-31T22:55:57.148 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67114.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139501.log 2024-08-31T22:55:57.148 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.144595.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104911.log 2024-08-31T22:55:57.148 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.139501.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.139501.log.gz 2024-08-31T22:55:57.149 INFO:teuthology.orchestra.run.smithi032.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.144595.log.gz 2024-08-31T22:55:57.149 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104632.log 2024-08-31T22:55:57.150 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.104911.log: 96.1% -- replaced with /var/log/ceph/cluster2-client.mirror.111149.log.gz 2024-08-31T22:55:57.150 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53437.log 2024-08-31T22:55:57.150 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.104632.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44604.log 2024-08-31T22:55:57.151 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53437.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41141.log 2024-08-31T22:55:57.151 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.44604.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31615.log 2024-08-31T22:55:57.151 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.41141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103157.log 2024-08-31T22:55:57.152 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.31615.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52782.log 2024-08-31T22:55:57.152 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.103157.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110705.log 2024-08-31T22:55:57.153 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.52782.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41488.log 2024-08-31T22:55:57.153 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.110705.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96840.log 2024-08-31T22:55:57.154 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.41488.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148237.log 2024-08-31T22:55:57.154 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96840.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110740.log 2024-08-31T22:55:57.154 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.148237.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163467.log 2024-08-31T22:55:57.154 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.110740.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106686.log 2024-08-31T22:55:57.155 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163467.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.163467.log.gz 2024-08-31T22:55:57.156 INFO:teuthology.orchestra.run.smithi032.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.148237.log.gz 2024-08-31T22:55:57.156 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53154.log 2024-08-31T22:55:57.156 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.106686.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50728.log 2024-08-31T22:55:57.157 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53154.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73936.log 2024-08-31T22:55:57.158 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50728.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.165476.log 2024-08-31T22:55:57.159 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.73936.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120623.log 2024-08-31T22:55:57.159 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.165476.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.165476.log.gz 2024-08-31T22:55:57.160 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34231.log 2024-08-31T22:55:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.120623.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91238.log 2024-08-31T22:55:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.34231.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116431.log 2024-08-31T22:55:57.162 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.91238.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55119.log 2024-08-31T22:55:57.162 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.116431.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67079.log 2024-08-31T22:55:57.162 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.55119.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82606.log 2024-08-31T22:55:57.162 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67079.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25243.log 2024-08-31T22:55:57.163 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82606.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43635.log 2024-08-31T22:55:57.163 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65622.log 2024-08-31T22:55:57.163 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43112.log.gz 2024-08-31T22:55:57.164 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.43635.log: /var/log/ceph/cluster2-client.admin.25243.log: 0.0% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63905.log.gz 2024-08-31T22:55:57.164 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.25243.log.gz 2024-08-31T22:55:57.165 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.88644.log.gz 2024-08-31T22:55:57.165 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118478.log 2024-08-31T22:55:57.166 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65622.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70758.log 2024-08-31T22:55:57.166 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.118478.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100541.log 2024-08-31T22:55:57.166 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.70758.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55712.log 2024-08-31T22:55:57.167 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.100541.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127427.log 2024-08-31T22:55:57.167 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.55712.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94317.log 2024-08-31T22:55:57.167 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.127427.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157703.log 2024-08-31T22:55:57.168 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94317.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.127427.log.gz 2024-08-31T22:55:57.168 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44697.log 2024-08-31T22:55:57.169 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157703.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92507.log 2024-08-31T22:55:57.170 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.44697.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.157703.log.gz 2024-08-31T22:55:57.170 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100018.log 2024-08-31T22:55:57.171 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136836.log 2024-08-31T22:55:57.171 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.92507.log: /var/log/ceph/cluster2-client.mirror.100018.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147739.log 2024-08-31T22:55:57.172 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103064.log 2024-08-31T22:55:57.173 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.136836.log: /var/log/ceph/cluster2-client.mirror.147739.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.147739.log.gz 2024-08-31T22:55:57.174 INFO:teuthology.orchestra.run.smithi032.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.136836.log.gz 2024-08-31T22:55:57.174 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71564.log 2024-08-31T22:55:57.175 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69044.log 2024-08-31T22:55:57.175 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.103064.log: /var/log/ceph/cluster2-client.mirror.71564.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47094.log 2024-08-31T22:55:57.176 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165013.log 2024-08-31T22:55:57.176 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.69044.log: /var/log/ceph/cluster2-client.mirror.47094.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50117.log.gz 2024-08-31T22:55:57.176 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50159.log.gz 2024-08-31T22:55:57.177 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140500.log 2024-08-31T22:55:57.177 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165013.log: 83.3% -- replaced with /var/log/ceph/cluster2-client.mirror.165013.log.gz 2024-08-31T22:55:57.177 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38936.log 2024-08-31T22:55:57.178 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.140500.log: 88.2%/var/log/ceph/cluster2-client.mirror.38936.log: -- replaced with /var/log/ceph/cluster2-client.mirror.53472.log.gz 2024-08-31T22:55:57.179 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.140500.log.gz 2024-08-31T22:55:57.179 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137401.log 2024-08-31T22:55:57.179 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33425.log 2024-08-31T22:55:57.179 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.53844.log.gz 2024-08-31T22:55:57.179 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.137401.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53530.log 2024-08-31T22:55:57.179 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56797.log.gz 2024-08-31T22:55:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33425.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.137401.log.gz 2024-08-31T22:55:57.180 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70948.log.gz 2024-08-31T22:55:57.181 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99828.log.gz 2024-08-31T22:55:57.181 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73188.log.gz 2024-08-31T22:55:57.182 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80238.log.gz 2024-08-31T22:55:57.182 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.106593.log.gz 2024-08-31T22:55:57.182 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38525.log.gz 2024-08-31T22:55:57.182 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38095.log.gz 2024-08-31T22:55:57.182 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44697.log.gz 2024-08-31T22:55:57.183 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.115776.log.gz 2024-08-31T22:55:57.183 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50728.log.gz 2024-08-31T22:55:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102820.log 2024-08-31T22:55:57.184 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78308.log 2024-08-31T22:55:57.184 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53530.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94503.log.gz 2024-08-31T22:55:57.184 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.102820.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49710.log 2024-08-31T22:55:57.185 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92170.log 2024-08-31T22:55:57.185 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.112585.log.gz 2024-08-31T22:55:57.185 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49710.log: /var/log/ceph/cluster2-client.mirror.78308.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161041.log 2024-08-31T22:55:57.186 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97832.log 2024-08-31T22:55:57.186 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.92170.log: /var/log/ceph/cluster1-client.mirror.161041.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.161041.log.gz 2024-08-31T22:55:57.187 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.95964.log.gz 2024-08-31T22:55:57.188 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119126.log.gz 2024-08-31T22:55:57.188 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48679.log 2024-08-31T22:55:57.189 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.97832.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.166033.log 2024-08-31T22:55:57.189 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69695.log.gz 2024-08-31T22:55:57.189 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48679.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31836.log 2024-08-31T22:55:57.190 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.82606.log.gz 2024-08-31T22:55:57.190 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.166033.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.166033.log.gz 2024-08-31T22:55:57.190 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43577.log 2024-08-31T22:55:57.191 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.31836.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72161.log.gz 2024-08-31T22:55:57.191 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.43577.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69978.log 2024-08-31T22:55:57.192 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43635.log.gz 2024-08-31T22:55:57.192 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32549.log 2024-08-31T22:55:57.192 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.69978.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78405.log 2024-08-31T22:55:57.193 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.32549.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46102.log.gz 2024-08-31T22:55:57.193 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46195.log.gz 2024-08-31T22:55:57.194 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47877.log 2024-08-31T22:55:57.194 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100762.log.gz 2024-08-31T22:55:57.194 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46381.log.gz 2024-08-31T22:55:57.195 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.78405.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46439.log.gz 2024-08-31T22:55:57.195 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46567.log.gz 2024-08-31T22:55:57.195 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143661.log 2024-08-31T22:55:57.196 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138200.log 2024-08-31T22:55:57.196 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.47877.log: /var/log/ceph/cluster2-client.mirror.143661.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.143661.log.gz 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.91274.log.gz 2024-08-31T22:55:57.196 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:57.197 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48400.log.gz 2024-08-31T22:55:57.197 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34603.log 2024-08-31T22:55:57.197 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.138200.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48679.log.gz 2024-08-31T22:55:57.198 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49710.log.gz 2024-08-31T22:55:57.198 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.49803.log.gz 2024-08-31T22:55:57.198 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.138200.log.gz 2024-08-31T22:55:57.199 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99607.log.gz 2024-08-31T22:55:57.199 INFO:teuthology.orchestra.run.smithi032.stderr:gzip 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69978.log.gz -5 2024-08-31T22:55:57.199 INFO:teuthology.orchestra.run.smithi032.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.162638.log 2024-08-31T22:55:57.200 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41488.log.gz 2024-08-31T22:55:57.200 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31836.log.gz 2024-08-31T22:55:57.200 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.34603.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41523.log.gz 2024-08-31T22:55:57.200 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32549.log.gz 2024-08-31T22:55:57.200 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33425.log.gz 2024-08-31T22:55:57.200 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35723.log.gz 2024-08-31T22:55:57.200 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.107934.log.gz 2024-08-31T22:55:57.202 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64502.log.gz 2024-08-31T22:55:57.202 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64595.log.gz 2024-08-31T22:55:57.203 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139215.log 2024-08-31T22:55:57.203 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162638.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.29512.log 2024-08-31T22:55:57.204 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% 86.7% -- replaced with /var/log/ceph/cluster2-client.mirror.109581.log.gz 2024-08-31T22:55:57.204 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.162638.log.gz 2024-08-31T22:55:57.204 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.139215.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.114046.log.gz 2024-08-31T22:55:57.204 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.139215.log.gz 2024-08-31T22:55:57.205 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119101.log 2024-08-31T22:55:57.205 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.0.29512.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.116055.log.gz 2024-08-31T22:55:57.205 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159676.log 2024-08-31T22:55:57.206 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119101.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68982.log 2024-08-31T22:55:57.206 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84474.log 2024-08-31T22:55:57.206 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.116617.log.gz 2024-08-31T22:55:57.207 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.68982.log: /var/log/ceph/cluster2-client.mirror.159676.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58386.log 2024-08-31T22:55:57.207 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.34603.log.gz 2024-08-31T22:55:57.207 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.159676.log.gz 2024-08-31T22:55:57.207 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.84474.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138936.log 2024-08-31T22:55:57.207 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58386.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52162.log 2024-08-31T22:55:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.138936.log: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.119604.log.gz 2024-08-31T22:55:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72723.log 2024-08-31T22:55:57.210 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.87427.log.gz 2024-08-31T22:55:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.52162.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.138936.log.gz 2024-08-31T22:55:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143719.log 2024-08-31T22:55:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.72723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115140.log 2024-08-31T22:55:57.211 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.143719.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76440.log.gz 2024-08-31T22:55:57.211 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.143719.log.gz 2024-08-31T22:55:57.211 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59789.log.gz 2024-08-31T22:55:57.211 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59731.log.gz 2024-08-31T22:55:57.212 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59634.log.gz 2024-08-31T22:55:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98615.log 2024-08-31T22:55:57.212 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.92170.log.gz 2024-08-31T22:55:57.213 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.115140.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45073.log 2024-08-31T22:55:57.213 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97832.log.gz 2024-08-31T22:55:57.214 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.98615.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46813.log 2024-08-31T22:55:57.214 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.45073.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116117.log 2024-08-31T22:55:57.214 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.46813.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64126.log 2024-08-31T22:55:57.214 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45538.log.gz 2024-08-31T22:55:57.214 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.54088.log.gz 2024-08-31T22:55:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.116117.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65622.log.gz 2024-08-31T22:55:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162098.log 2024-08-31T22:55:57.216 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64126.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29079.log 2024-08-31T22:55:57.216 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162098.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.163402.log 2024-08-31T22:55:57.217 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.29079.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68982.log.gz 2024-08-31T22:55:57.218 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50226.log.gz 2024-08-31T22:55:57.218 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.53751.log.gz 2024-08-31T22:55:57.218 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53530.log.gz 2024-08-31T22:55:57.218 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.62658.log.gz 2024-08-31T22:55:57.218 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71471.log.gz 2024-08-31T22:55:57.219 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32677.log 2024-08-31T22:55:57.219 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.163402.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.163402.log.gz 2024-08-31T22:55:57.219 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73808.log.gz 2024-08-31T22:55:57.219 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160375.log 2024-08-31T22:55:57.219 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.32677.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102223.log.gz 2024-08-31T22:55:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161108.log 2024-08-31T22:55:57.220 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104911.log.gz 2024-08-31T22:55:57.221 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.160375.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.104725.log.gz 2024-08-31T22:55:57.221 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.162098.log.gz 2024-08-31T22:55:57.221 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44848.log.gz 2024-08-31T22:55:57.221 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.160375.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.102820.log.gz 2024-08-31T22:55:57.221 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:57.221 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62498.log 2024-08-31T22:55:57.221 INFO:teuthology.orchestra.run.smithi032.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.29079.log.gz 2024-08-31T22:55:57.222 INFO:teuthology.orchestra.run.smithi032.stderr: 69.6% -- replaced with /var/log/ceph/cluster1-client.admin.29096.log.gz 2024-08-31T22:55:57.222 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161108.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119101.log.gz 2024-08-31T22:55:57.222 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.161108.log.gz 2024-08-31T22:55:57.222 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30340.log 2024-08-31T22:55:57.222 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84474.log.gz 2024-08-31T22:55:57.223 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62498.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39777.log 2024-08-31T22:55:57.223 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.30340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104256.log 2024-08-31T22:55:57.223 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39777.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95344.log 2024-08-31T22:55:57.223 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100634.log.gz 2024-08-31T22:55:57.224 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.104256.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156618.log 2024-08-31T22:55:57.224 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.95344.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159767.log 2024-08-31T22:55:57.224 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156618.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51100.log 2024-08-31T22:55:57.225 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159767.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.116117.log.gz 2024-08-31T22:55:57.227 INFO:teuthology.orchestra.run.smithi032.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.159767.log.gz 2024-08-31T22:55:57.227 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43577.log.gz 2024-08-31T22:55:57.227 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.156618.log.gz 2024-08-31T22:55:57.228 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149653.log 2024-08-31T22:55:57.228 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.89706.log.gz 2024-08-31T22:55:57.228 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51100.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46813.log.gz 2024-08-31T22:55:57.228 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46908.log.gz 2024-08-31T22:55:57.228 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47877.log.gz 2024-08-31T22:55:57.228 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82571.log 2024-08-31T22:55:57.229 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.149653.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109992.log 2024-08-31T22:55:57.230 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58386.log.gz 2024-08-31T22:55:57.230 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82571.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.149653.log.gz 2024-08-31T22:55:57.230 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56704.log 2024-08-31T22:55:57.230 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.109992.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54313.log 2024-08-31T22:55:57.230 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.56704.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115108.log 2024-08-31T22:55:57.231 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.54313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159654.log 2024-08-31T22:55:57.231 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.115108.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110364.log 2024-08-31T22:55:57.232 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159654.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34138.log 2024-08-31T22:55:57.232 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.110364.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.159654.log.gz 2024-08-31T22:55:57.232 INFO:teuthology.orchestra.run.smithi032.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.151225.log.gz 2024-08-31T22:55:57.232 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73153.log 2024-08-31T22:55:57.233 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.34138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91477.log 2024-08-31T22:55:57.233 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35072.log 2024-08-31T22:55:57.233 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.73153.log: /var/log/ceph/cluster1-client.admin.91477.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80459.log 2024-08-31T22:55:57.234 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.35072.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164415.log 2024-08-31T22:55:57.234 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111210.log 2024-08-31T22:55:57.235 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.80459.log: /var/log/ceph/cluster2-client.mirror.164415.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145967.log 2024-08-31T22:55:57.235 INFO:teuthology.orchestra.run.smithi032.stderr: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.164415.log.gz/var/log/ceph/cluster1-client.admin.111210.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53061.log 2024-08-31T22:55:57.235 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:57.235 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.145967.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157945.log 2024-08-31T22:55:57.235 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53061.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75190.log 2024-08-31T22:55:57.237 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157945.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.157945.log.gz 2024-08-31T22:55:57.237 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88117.log 2024-08-31T22:55:57.238 INFO:teuthology.orchestra.run.smithi032.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.145967.log.gz 2024-08-31T22:55:57.238 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75190.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73339.log 2024-08-31T22:55:57.238 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.88117.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75214.log 2024-08-31T22:55:57.238 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.73339.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90636.log 2024-08-31T22:55:57.240 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.75214.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106159.log 2024-08-31T22:55:57.240 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.90636.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64091.log 2024-08-31T22:55:57.240 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.106159.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137854.log 2024-08-31T22:55:57.240 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64091.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57359.log 2024-08-31T22:55:57.240 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111172.log 2024-08-31T22:55:57.240 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.137854.log: /var/log/ceph/cluster2-client.mirror.57359.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158946.log 2024-08-31T22:55:57.240 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137854.log.gz 2024-08-31T22:55:57.241 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.111172.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76347.log 2024-08-31T22:55:57.241 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158946.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81300.log 2024-08-31T22:55:57.243 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.76347.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107806.log 2024-08-31T22:55:57.243 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.81300.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29369.log 2024-08-31T22:55:57.243 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.158946.log.gz 2024-08-31T22:55:57.243 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107806.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88958.log 2024-08-31T22:55:57.243 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.29369.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32642.log 2024-08-31T22:55:57.243 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.88958.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51379.log 2024-08-31T22:55:57.243 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.32642.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91388.log 2024-08-31T22:55:57.243 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51379.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101886.log 2024-08-31T22:55:57.243 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.91388.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149359.log 2024-08-31T22:55:57.244 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101886.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52162.log.gz 2024-08-31T22:55:57.245 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31929.log.gz 2024-08-31T22:55:57.245 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32677.log.gz 2024-08-31T22:55:57.245 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41176.log.gz 2024-08-31T22:55:57.245 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32642.log.gz 2024-08-31T22:55:57.246 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.109957.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.109992.log.gz 2024-08-31T22:55:57.246 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:57.247 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64277.log 2024-08-31T22:55:57.247 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114081.log.gz 2024-08-31T22:55:57.247 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.149359.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40769.log 2024-08-31T22:55:57.248 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64277.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148125.log 2024-08-31T22:55:57.248 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.40769.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117272.log 2024-08-31T22:55:57.248 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.148125.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.155812.log 2024-08-31T22:55:57.249 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.117272.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.148125.log.gz 2024-08-31T22:55:57.249 INFO:teuthology.orchestra.run.smithi032.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.149359.log.gz 2024-08-31T22:55:57.252 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67827.log 2024-08-31T22:55:57.252 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.155812.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95871.log 2024-08-31T22:55:57.253 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67827.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.155812.log.gz 2024-08-31T22:55:57.253 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43542.log 2024-08-31T22:55:57.253 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.95871.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29014.log 2024-08-31T22:55:57.254 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93852.log 2024-08-31T22:55:57.254 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.43542.log: /var/log/ceph/cluster1-client.admin.29014.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82513.log 2024-08-31T22:55:57.254 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29014.log.gz 2024-08-31T22:55:57.254 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.93852.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63099.log 2024-08-31T22:55:57.256 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82513.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86528.log 2024-08-31T22:55:57.256 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63099.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92976.log 2024-08-31T22:55:57.256 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86528.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39649.log 2024-08-31T22:55:57.256 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149688.log 2024-08-31T22:55:57.256 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.92976.log: /var/log/ceph/cluster2-client.mirror.39649.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63157.log 2024-08-31T22:55:57.256 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.149688.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157330.log 2024-08-31T22:55:57.257 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1%/var/log/ceph/cluster2-client.mirror.63157.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137668.log 2024-08-31T22:55:57.257 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.149688.log.gz 2024-08-31T22:55:57.258 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157330.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85935.log 2024-08-31T22:55:57.258 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.137668.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38339.log 2024-08-31T22:55:57.258 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.157330.log.gz 2024-08-31T22:55:57.258 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.85935.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137668.log.gz 2024-08-31T22:55:57.260 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28774.log 2024-08-31T22:55:57.261 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.38339.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30247.log 2024-08-31T22:55:57.261 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.28774.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92077.log 2024-08-31T22:55:57.261 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28774.log.gz 2024-08-31T22:55:57.261 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.30247.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115497.log 2024-08-31T22:55:57.261 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.92077.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166423.log 2024-08-31T22:55:57.261 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.90264.log.gz 2024-08-31T22:55:57.261 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.115497.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159483.log 2024-08-31T22:55:57.261 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107527.log 2024-08-31T22:55:57.261 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.166423.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166423.log.gz 2024-08-31T22:55:57.261 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159483.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141935.log 2024-08-31T22:55:57.261 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107527.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164992.log 2024-08-31T22:55:57.262 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.141935.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.113767.log.gz 2024-08-31T22:55:57.262 INFO:teuthology.orchestra.run.smithi032.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.159483.log.gz 2024-08-31T22:55:57.263 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88454.log 2024-08-31T22:55:57.263 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164992.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163871.log 2024-08-31T22:55:57.263 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.88454.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76254.log.gz 2024-08-31T22:55:57.264 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76595.log.gz 2024-08-31T22:55:57.264 INFO:teuthology.orchestra.run.smithi032.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118141.log.gz 2024-08-31T22:55:57.264 INFO:teuthology.orchestra.run.smithi032.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.164992.log.gz 2024-08-31T22:55:57.265 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84718.log 2024-08-31T22:55:57.265 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163871.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94786.log 2024-08-31T22:55:57.268 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.84718.log: 96.8% -- replaced with /var/log/ceph/cluster2-client.mirror.141935.log.gz 2024-08-31T22:55:57.268 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.80610.log.gz 2024-08-31T22:55:57.269 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.95158.log.gz 2024-08-31T22:55:57.269 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159978.log 2024-08-31T22:55:57.270 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63343.log 2024-08-31T22:55:57.270 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.98615.log.gz 2024-08-31T22:55:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159978.log: 96.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103791.log.gz 2024-08-31T22:55:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50411.log 2024-08-31T22:55:57.271 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.159978.log.gz 2024-08-31T22:55:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63343.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148615.log 2024-08-31T22:55:57.276 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.50411.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30340.log.gz 2024-08-31T22:55:57.278 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34138.log.gz 2024-08-31T22:55:57.278 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34173.log.gz 2024-08-31T22:55:57.281 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.78308.log.gz 2024-08-31T22:55:57.281 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.78405.log.gz 2024-08-31T22:55:57.281 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57173.log.gz 2024-08-31T22:55:57.282 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158826.log 2024-08-31T22:55:57.282 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.148615.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120939.log 2024-08-31T22:55:57.283 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62971.log.gz 2024-08-31T22:55:57.283 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63157.log.gz 2024-08-31T22:55:57.283 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.63099.log.gz 2024-08-31T22:55:57.284 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158826.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45073.log.gz 2024-08-31T22:55:57.284 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.88454.log.gz 2024-08-31T22:55:57.284 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.158826.log.gz 2024-08-31T22:55:57.284 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63343.log.gz 2024-08-31T22:55:57.284 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69044.log.gz 2024-08-31T22:55:57.284 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69881.log 2024-08-31T22:55:57.284 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65025.log.gz 2024-08-31T22:55:57.284 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.120939.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107283.log 2024-08-31T22:55:57.285 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.69881.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98987.log 2024-08-31T22:55:57.285 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31371.log 2024-08-31T22:55:57.285 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.98987.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.148615.log.gz 2024-08-31T22:55:57.286 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.51100.log.gz 2024-08-31T22:55:57.286 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56704.log.gz 2024-08-31T22:55:57.287 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62498.log.gz 2024-08-31T22:55:57.288 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64277.log.gz 2024-08-31T22:55:57.288 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.102630.log.gz 2024-08-31T22:55:57.288 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.104632.log.gz 2024-08-31T22:55:57.288 INFO:teuthology.orchestra.run.smithi032.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125944.log.gz 2024-08-31T22:55:57.289 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.154063.log 2024-08-31T22:55:57.289 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.31371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143811.log 2024-08-31T22:55:57.290 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.115108.log.gz 2024-08-31T22:55:57.291 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.115140.log.gz 2024-08-31T22:55:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62029.log 2024-08-31T22:55:57.291 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110705.log.gz 2024-08-31T22:55:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.143811.log: /var/log/ceph/cluster2-client.mirror.154063.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122008.log 2024-08-31T22:55:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62029.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119080.log.gz 2024-08-31T22:55:57.291 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.154063.log.gz 2024-08-31T22:55:57.292 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164739.log 2024-08-31T22:55:57.292 INFO:teuthology.orchestra.run.smithi032.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.120939.log.gz 2024-08-31T22:55:57.292 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.122008.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31239.log.gz 2024-08-31T22:55:57.292 INFO:teuthology.orchestra.run.smithi032.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122008.log.gz 2024-08-31T22:55:57.292 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.50411.log.gz 2024-08-31T22:55:57.293 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31371.log.gz 2024-08-31T22:55:57.293 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52724.log 2024-08-31T22:55:57.293 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159239.log 2024-08-31T22:55:57.294 INFO:teuthology.orchestra.run.smithi032.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.143811.log.gz 2024-08-31T22:55:57.294 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164739.log: /var/log/ceph/cluster2-client.mirror.52724.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.164739.log.gz 2024-08-31T22:55:57.294 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29215.log 2024-08-31T22:55:57.295 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159239.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105973.log 2024-08-31T22:55:57.295 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.29215.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98580.log 2024-08-31T22:55:57.295 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.159239.log.gz 2024-08-31T22:55:57.295 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105973.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.94096.log.gz 2024-08-31T22:55:57.295 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37068.log 2024-08-31T22:55:57.296 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.98580.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.94786.log.gz 2024-08-31T22:55:57.296 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100541.log.gz 2024-08-31T22:55:57.297 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.89388.log.gz 2024-08-31T22:55:57.297 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.86904.log.gz 2024-08-31T22:55:57.298 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.admin.111210.log.gz 2024-08-31T22:55:57.298 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82571.log.gz 2024-08-31T22:55:57.298 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.105973.log.gz 2024-08-31T22:55:57.298 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.89485.log.gz 2024-08-31T22:55:57.299 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.82664.log.gz 2024-08-31T22:55:57.299 INFO:teuthology.orchestra.run.smithi032.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103702.log.gz 2024-08-31T22:55:57.299 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86962.log 2024-08-31T22:55:57.299 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.37068.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160282.log 2024-08-31T22:55:57.299 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86962.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45784.log 2024-08-31T22:55:57.300 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.160282.log: /var/log/ceph/cluster2-client.mirror.45784.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.160282.log.gz 2024-08-31T22:55:57.301 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99142.log 2024-08-31T22:55:57.301 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34080.log 2024-08-31T22:55:57.302 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.99142.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118188.log 2024-08-31T22:55:57.302 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.34080.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119277.log 2024-08-31T22:55:57.302 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113019.log 2024-08-31T22:55:57.302 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118188.log: /var/log/ceph/cluster2-client.mirror.119277.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105508.log 2024-08-31T22:55:57.303 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.113019.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108124.log 2024-08-31T22:55:57.303 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162053.log 2024-08-31T22:55:57.303 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105508.log: /var/log/ceph/cluster2-client.mirror.108124.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36413.log 2024-08-31T22:55:57.304 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65529.log 2024-08-31T22:55:57.304 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52724.log.gz 2024-08-31T22:55:57.304 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162053.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55154.log.gz 2024-08-31T22:55:57.305 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.36413.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.162053.log.gz 2024-08-31T22:55:57.305 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69881.log.gz 2024-08-31T22:55:57.307 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53247.log 2024-08-31T22:55:57.307 INFO:teuthology.orchestra.run.smithi032.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127450.log.gz 2024-08-31T22:55:57.307 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65529.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158755.log 2024-08-31T22:55:57.307 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37068.log.gz 2024-08-31T22:55:57.309 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88865.log 2024-08-31T22:55:57.309 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53247.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.163347.log 2024-08-31T22:55:57.309 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158755.log: /var/log/ceph/cluster2-client.mirror.88865.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40769.log.gz 2024-08-31T22:55:57.309 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.158755.log.gz 2024-08-31T22:55:57.309 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161783.log 2024-08-31T22:55:57.311 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.163347.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.164023.log 2024-08-31T22:55:57.311 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161783.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79955.log 2024-08-31T22:55:57.311 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.163347.log.gz 2024-08-31T22:55:57.311 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.161783.log.gz 2024-08-31T22:55:57.311 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28291.log 2024-08-31T22:55:57.311 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.164023.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41141.log.gz 2024-08-31T22:55:57.311 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.79955.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69230.log 2024-08-31T22:55:57.311 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0%/var/log/ceph/cluster2-client.admin.28291.log: -- replaced with /var/log/ceph/cluster1-client.mirror.164023.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68203.log 2024-08-31T22:55:57.311 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28291.log.gz 2024-08-31T22:55:57.312 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:57.312 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75564.log 2024-08-31T22:55:57.312 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.69230.log: /var/log/ceph/cluster2-client.mirror.68203.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161536.log 2024-08-31T22:55:57.313 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75564.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152375.log 2024-08-31T22:55:57.313 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161536.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104384.log 2024-08-31T22:55:57.313 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.161536.log.gz 2024-08-31T22:55:57.313 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.92077.log.gz 2024-08-31T22:55:57.313 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.152375.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.107283.log.gz 2024-08-31T22:55:57.327 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102878.log 2024-08-31T22:55:57.327 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.104384.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42550.log 2024-08-31T22:55:57.328 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150780.log 2024-08-31T22:55:57.328 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.42550.log: /var/log/ceph/cluster2-client.mirror.102878.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123192.log 2024-08-31T22:55:57.328 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.150780.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.150780.log.gz 2024-08-31T22:55:57.329 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.86121.log.gz 2024-08-31T22:55:57.339 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.83691.log.gz 2024-08-31T22:55:57.339 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.91608.log.gz 2024-08-31T22:55:57.339 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.149787.log.gz 2024-08-31T22:55:57.340 INFO:teuthology.orchestra.run.smithi032.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.122707.log.gz 2024-08-31T22:55:57.340 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87276.log 2024-08-31T22:55:57.341 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.96840.log.gz 2024-08-31T22:55:57.341 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143340.log 2024-08-31T22:55:57.341 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.123192.log: /var/log/ceph/cluster2-client.mirror.87276.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105601.log 2024-08-31T22:55:57.342 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.143340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36041.log 2024-08-31T22:55:57.342 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91049.log 2024-08-31T22:55:57.342 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61564.log 2024-08-31T22:55:57.342 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.36041.log: /var/log/ceph/cluster2-client.mirror.91049.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74087.log 2024-08-31T22:55:57.342 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/cluster2-client.mirror.61564.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119038.log 2024-08-31T22:55:57.344 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.74087.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.143340.log.gz 2024-08-31T22:55:57.345 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118075.log 2024-08-31T22:55:57.345 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147774.log 2024-08-31T22:55:57.345 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119038.log: /var/log/ceph/cluster1-client.mirror.118075.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30097.log 2024-08-31T22:55:57.345 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.147774.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115648.log 2024-08-31T22:55:57.345 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.30097.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30212.log 2024-08-31T22:55:57.345 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.147774.log.gz 2024-08-31T22:55:57.345 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.115648.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100727.log 2024-08-31T22:55:57.345 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.30212.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54964.log 2024-08-31T22:55:57.345 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.100727.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136965.log 2024-08-31T22:55:57.345 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.54964.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28549.log 2024-08-31T22:55:57.346 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.136965.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84381.log 2024-08-31T22:55:57.346 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.28549.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.29577.log 2024-08-31T22:55:57.346 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.136965.log.gz 2024-08-31T22:55:57.346 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28549.log.gz 2024-08-31T22:55:57.347 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.84381.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51193.log 2024-08-31T22:55:57.347 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.0.29577.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81544.log 2024-08-31T22:55:57.348 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118929.log 2024-08-31T22:55:57.348 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62626.log 2024-08-31T22:55:57.348 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.81544.log: /var/log/ceph/cluster2-client.mirror.118929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60851.log 2024-08-31T22:55:57.349 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62626.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110236.log 2024-08-31T22:55:57.349 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146664.log 2024-08-31T22:55:57.349 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60851.log: /var/log/ceph/cluster2-client.mirror.110236.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142996.log 2024-08-31T22:55:57.349 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/cluster1-client.mirror.146664.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39684.log 2024-08-31T22:55:57.350 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.142996.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82947.log 2024-08-31T22:55:57.350 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.146664.log.gz 2024-08-31T22:55:57.355 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39684.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94224.log 2024-08-31T22:55:57.355 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82947.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78928.log 2024-08-31T22:55:57.355 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94224.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61413.log 2024-08-31T22:55:57.356 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.78928.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106872.log 2024-08-31T22:55:57.356 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61413.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117086.log 2024-08-31T22:55:57.356 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.106872.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102258.log 2024-08-31T22:55:57.357 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.117086.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74122.log 2024-08-31T22:55:57.357 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.102258.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81823.log 2024-08-31T22:55:57.357 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.74122.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101010.log 2024-08-31T22:55:57.358 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.81823.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115233.log 2024-08-31T22:55:57.358 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101010.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79056.log 2024-08-31T22:55:57.359 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.115233.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81916.log 2024-08-31T22:55:57.359 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.79056.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76816.log 2024-08-31T22:55:57.359 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.81916.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.164907.log 2024-08-31T22:55:57.360 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.76816.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163742.log 2024-08-31T22:55:57.360 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.164907.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53937.log 2024-08-31T22:55:57.360 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.164907.log.gz 2024-08-31T22:55:57.360 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163742.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.163742.log.gz 2024-08-31T22:55:57.371 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53096.log 2024-08-31T22:55:57.371 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53937.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92414.log 2024-08-31T22:55:57.371 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53096.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74684.log 2024-08-31T22:55:57.372 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76068.log 2024-08-31T22:55:57.372 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.92414.log: /var/log/ceph/cluster2-client.mirror.74684.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94375.log 2024-08-31T22:55:57.373 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.76068.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156470.log 2024-08-31T22:55:57.373 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30495.log 2024-08-31T22:55:57.374 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94375.log: /var/log/ceph/cluster2-client.mirror.156470.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56553.log 2024-08-31T22:55:57.374 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.30495.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42515.log 2024-08-31T22:55:57.375 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107376.log 2024-08-31T22:55:57.375 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.156470.log.gz 2024-08-31T22:55:57.375 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.56553.log: /var/log/ceph/cluster2-client.mirror.42515.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160253.log 2024-08-31T22:55:57.375 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60816.log 2024-08-31T22:55:57.376 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107376.log: /var/log/ceph/cluster2-client.mirror.160253.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.88303.log.gz 2024-08-31T22:55:57.376 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115834.log 2024-08-31T22:55:57.376 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60816.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165307.log 2024-08-31T22:55:57.377 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.115834.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65529.log.gz 2024-08-31T22:55:57.377 INFO:teuthology.orchestra.run.smithi032.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.160253.log.gz 2024-08-31T22:55:57.378 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45631.log.gz 2024-08-31T22:55:57.378 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45784.log.gz 2024-08-31T22:55:57.378 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140597.log 2024-08-31T22:55:57.379 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165307.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51941.log 2024-08-31T22:55:57.379 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.140597.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.165307.log.gz 2024-08-31T22:55:57.379 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76498.log 2024-08-31T22:55:57.379 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.140597.log.gz 2024-08-31T22:55:57.379 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56518.log 2024-08-31T22:55:57.380 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51941.log: /var/log/ceph/cluster2-client.mirror.76498.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108554.log 2024-08-31T22:55:57.380 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.56518.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.164720.log 2024-08-31T22:55:57.380 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.108554.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157167.log 2024-08-31T22:55:57.381 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.164720.log: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.164720.log.gz 2024-08-31T22:55:57.381 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139835.log 2024-08-31T22:55:57.381 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157167.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.157167.log.gz 2024-08-31T22:55:57.382 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62157.log 2024-08-31T22:55:57.387 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.139835.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88210.log 2024-08-31T22:55:57.387 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44790.log 2024-08-31T22:55:57.388 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62157.log: /var/log/ceph/cluster2-client.mirror.88210.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156792.log 2024-08-31T22:55:57.388 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62405.log 2024-08-31T22:55:57.388 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.44790.log: /var/log/ceph/cluster2-client.mirror.156792.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95778.log 2024-08-31T22:55:57.390 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62405.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.156792.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109058.log 2024-08-31T22:55:57.390 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:57.391 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.95778.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70758.log.gz 2024-08-31T22:55:57.391 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71657.log.gz 2024-08-31T22:55:57.392 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79618.log 2024-08-31T22:55:57.393 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113709.log 2024-08-31T22:55:57.393 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.109058.log: /var/log/ceph/cluster2-client.mirror.79618.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70071.log 2024-08-31T22:55:57.394 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.70071.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162616.log 2024-08-31T22:55:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72630.log 2024-08-31T22:55:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.113709.log: /var/log/ceph/cluster1-client.mirror.162616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30066.log 2024-08-31T22:55:57.395 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.162616.log.gz 2024-08-31T22:55:57.396 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.72630.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90229.log 2024-08-31T22:55:57.397 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.30066.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.102878.log.gz 2024-08-31T22:55:57.397 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.104384.log.gz 2024-08-31T22:55:57.397 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37626.log.gz 2024-08-31T22:55:57.397 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103250.log 2024-08-31T22:55:57.398 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.30066.log.gz 2024-08-31T22:55:57.398 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.90229.log: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.29215.log.gz 2024-08-31T22:55:57.398 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78273.log 2024-08-31T22:55:57.399 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.103250.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.113019.log.gz 2024-08-31T22:55:57.399 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86869.log 2024-08-31T22:55:57.400 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.78273.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84718.log.gz 2024-08-31T22:55:57.400 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116338.log.gz 2024-08-31T22:55:57.400 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.105601.log.gz 2024-08-31T22:55:57.402 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159725.log 2024-08-31T22:55:57.403 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.165845.log 2024-08-31T22:55:57.404 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86869.log: /var/log/ceph/cluster2-client.mirror.159725.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106031.log 2024-08-31T22:55:57.404 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.165845.log: 86.9% 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.165845.log.gz 2024-08-31T22:55:57.405 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.159725.log.gz 2024-08-31T22:55:57.405 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136544.log 2024-08-31T22:55:57.405 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.106031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29038.log 2024-08-31T22:55:57.406 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.136544.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.136544.log.gz 2024-08-31T22:55:57.407 INFO:teuthology.orchestra.run.smithi032.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.90538.log.gz 2024-08-31T22:55:57.407 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161221.log 2024-08-31T22:55:57.407 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.29038.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29038.log.gz 2024-08-31T22:55:57.407 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119768.log 2024-08-31T22:55:57.409 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161221.log: 86.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55057.log 2024-08-31T22:55:57.409 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.161221.log.gz 2024-08-31T22:55:57.409 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119768.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120092.log 2024-08-31T22:55:57.409 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157421.log 2024-08-31T22:55:57.410 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.55057.log: /var/log/ceph/cluster2-client.mirror.120092.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33204.log 2024-08-31T22:55:57.410 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.117272.log.gz 2024-08-31T22:55:57.410 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157421.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92918.log 2024-08-31T22:55:57.411 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33204.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145203.log 2024-08-31T22:55:57.412 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.92918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.29645.log 2024-08-31T22:55:57.413 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.145203.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.157421.log.gz 2024-08-31T22:55:57.415 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166629.log 2024-08-31T22:55:57.415 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.1.29645.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91949.log 2024-08-31T22:55:57.416 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.166629.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166629.log.gz 2024-08-31T22:55:57.416 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156743.log 2024-08-31T22:55:57.417 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.91949.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70478.log 2024-08-31T22:55:57.417 INFO:teuthology.orchestra.run.smithi032.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.145203.log.gz 2024-08-31T22:55:57.418 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156299.log 2024-08-31T22:55:57.418 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156743.log: /var/log/ceph/cluster2-client.mirror.70478.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67920.log.gz 2024-08-31T22:55:57.419 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.156743.log.gz 2024-08-31T22:55:57.419 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41412.log 2024-08-31T22:55:57.419 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68238.log.gz 2024-08-31T22:55:57.419 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156299.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41860.log 2024-08-31T22:55:57.420 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68668.log.gz 2024-08-31T22:55:57.420 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68145.log.gz 2024-08-31T22:55:57.420 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68203.log.gz 2024-08-31T22:55:57.420 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.41412.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90078.log 2024-08-31T22:55:57.421 INFO:teuthology.orchestra.run.smithi032.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.156299.log.gz 2024-08-31T22:55:57.421 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.41860.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52875.log 2024-08-31T22:55:57.422 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.90078.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55433.log 2024-08-31T22:55:57.422 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.52875.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68575.log.gz 2024-08-31T22:55:57.423 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102316.log 2024-08-31T22:55:57.423 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.55433.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.155054.log 2024-08-31T22:55:57.424 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.102316.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56553.log.gz 2024-08-31T22:55:57.424 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56518.log.gz 2024-08-31T22:55:57.424 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56460.log.gz 2024-08-31T22:55:57.424 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41860.log.gz 2024-08-31T22:55:57.424 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77153.log 2024-08-31T22:55:57.425 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.155054.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117237.log 2024-08-31T22:55:57.425 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77153.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122452.log 2024-08-31T22:55:57.426 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.117237.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165286.log 2024-08-31T22:55:57.426 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.122452.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38936.log.gz 2024-08-31T22:55:57.427 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52875.log.gz 2024-08-31T22:55:57.427 INFO:teuthology.orchestra.run.smithi032.stderr: 82.8% -- replaced with /var/log/ceph/cluster2-client.mirror.122452.log.gz 2024-08-31T22:55:57.427 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160191.log 2024-08-31T22:55:57.428 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165286.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.165286.log.gz 2024-08-31T22:55:57.428 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55057.log.gz 2024-08-31T22:55:57.429 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.82013.log.gz 2024-08-31T22:55:57.429 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70071.log.gz 2024-08-31T22:55:57.431 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.107376.log.gz 2024-08-31T22:55:57.431 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.108554.log.gz 2024-08-31T22:55:57.432 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109058.log.gz 2024-08-31T22:55:57.432 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.36041.log.gz 2024-08-31T22:55:57.432 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36413.log.gz 2024-08-31T22:55:57.436 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.81451.log.gz 2024-08-31T22:55:57.436 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114736.log.gz 2024-08-31T22:55:57.436 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34324.log.gz 2024-08-31T22:55:57.436 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.90078.log.gz 2024-08-31T22:55:57.438 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76347.log.gz 2024-08-31T22:55:57.439 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.87276.log.gz 2024-08-31T22:55:57.439 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32921.log 2024-08-31T22:55:57.439 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.160191.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.92507.log.gz 2024-08-31T22:55:57.440 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.160191.log.gz 2024-08-31T22:55:57.440 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69265.log 2024-08-31T22:55:57.443 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.32921.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89671.log 2024-08-31T22:55:57.443 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88865.log.gz 2024-08-31T22:55:57.444 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.69265.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117950.log 2024-08-31T22:55:57.444 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89671.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138165.log 2024-08-31T22:55:57.445 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.117950.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56332.log 2024-08-31T22:55:57.446 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.138165.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.138165.log.gz 2024-08-31T22:55:57.447 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109302.log 2024-08-31T22:55:57.447 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.129671.log.gz/var/log/ceph/cluster2-client.mirror.56332.log: 2024-08-31T22:55:57.447 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81017.log 2024-08-31T22:55:57.447 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.109302.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117685.log 2024-08-31T22:55:57.448 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.165545.log 2024-08-31T22:55:57.448 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.81017.log: /var/log/ceph/cluster2-client.mirror.117685.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36006.log 2024-08-31T22:55:57.449 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101661.log 2024-08-31T22:55:57.449 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.165545.log: /var/log/ceph/cluster2-client.mirror.36006.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.165545.log.gz 2024-08-31T22:55:57.449 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146731.log 2024-08-31T22:55:57.450 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94972.log 2024-08-31T22:55:57.450 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101661.log: /var/log/ceph/cluster2-client.mirror.146731.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156279.log 2024-08-31T22:55:57.451 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93162.log 2024-08-31T22:55:57.452 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156279.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.80366.log.gz 2024-08-31T22:55:57.452 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.156279.log.gz 2024-08-31T22:55:57.452 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150710.log 2024-08-31T22:55:57.453 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.93162.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39215.log 2024-08-31T22:55:57.454 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.150710.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.150710.log.gz 2024-08-31T22:55:57.454 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98708.log 2024-08-31T22:55:57.455 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39215.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29956.log 2024-08-31T22:55:57.458 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.98708.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91342.log.gz 2024-08-31T22:55:57.460 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69637.log.gz 2024-08-31T22:55:57.463 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53061.log.gz 2024-08-31T22:55:57.463 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69265.log.gz 2024-08-31T22:55:57.463 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.60816.log.gz 2024-08-31T22:55:57.463 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64405.log.gz 2024-08-31T22:55:57.463 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64126.log.gz 2024-08-31T22:55:57.463 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.106686.log.gz 2024-08-31T22:55:57.464 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166345.log 2024-08-31T22:55:57.464 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.59076.log.gz 2024-08-31T22:55:57.464 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30681.log.gz 2024-08-31T22:55:57.464 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.29956.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100018.log.gz 2024-08-31T22:55:57.464 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94693.log 2024-08-31T22:55:57.464 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.104070.log.gz 2024-08-31T22:55:57.465 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.103250.log.gz 2024-08-31T22:55:57.465 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.166345.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166345.log.gz 2024-08-31T22:55:57.465 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43077.log 2024-08-31T22:55:57.467 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94693.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106031.log.gz 2024-08-31T22:55:57.467 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.105566.log.gz 2024-08-31T22:55:57.467 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.49896.log.gz 2024-08-31T22:55:57.467 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.86869.log.gz 2024-08-31T22:55:57.467 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51379.log.gz 2024-08-31T22:55:57.467 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.89671.log.gz 2024-08-31T22:55:57.467 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118568.log 2024-08-31T22:55:57.468 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.43077.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144575.log 2024-08-31T22:55:57.481 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.118568.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55119.log.gz 2024-08-31T22:55:57.482 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32921.log.gz 2024-08-31T22:55:57.482 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.35572.log.gz 2024-08-31T22:55:57.482 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52410.log.gz 2024-08-31T22:55:57.482 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53003.log.gz 2024-08-31T22:55:57.487 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163372.log 2024-08-31T22:55:57.490 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.144575.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.144575.log.gz 2024-08-31T22:55:57.490 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107899.log.gz 2024-08-31T22:55:57.495 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42550.log.gz 2024-08-31T22:55:57.497 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75882.log.gz 2024-08-31T22:55:57.497 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117950.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.80459.log.gz 2024-08-31T22:55:57.497 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:57.498 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.92356.log.gz 2024-08-31T22:55:57.498 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.95344.log.gz 2024-08-31T22:55:57.522 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43077.log.gz 2024-08-31T22:55:57.522 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65060.log.gz 2024-08-31T22:55:57.522 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57359.log.gz 2024-08-31T22:55:57.522 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61413.log.gz 2024-08-31T22:55:57.523 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62029.log.gz 2024-08-31T22:55:57.523 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64091.log.gz 2024-08-31T22:55:57.524 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73936.log.gz 2024-08-31T22:55:57.524 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.115497.log.gz 2024-08-31T22:55:57.525 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.115648.log.gz 2024-08-31T22:55:57.529 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67304.log.gz 2024-08-31T22:55:57.534 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31615.log.gz 2024-08-31T22:55:57.534 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39777.log.gz 2024-08-31T22:55:57.535 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.94693.log.gz 2024-08-31T22:55:57.536 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.95871.log.gz 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.101103.log.gz 2024-08-31T22:55:57.536 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:57.537 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.112833.log.gz 2024-08-31T22:55:57.537 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89799.log 2024-08-31T22:55:57.538 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163372.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99047.log.gz 2024-08-31T22:55:57.539 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.98987.log.gz 2024-08-31T22:55:57.540 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.37312.log.gz 2024-08-31T22:55:57.540 INFO:teuthology.orchestra.run.smithi032.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.163372.log.gz 2024-08-31T22:55:57.540 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43484.log.gz 2024-08-31T22:55:57.540 INFO:teuthology.orchestra.run.smithi032.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.115169.log.gz 2024-08-31T22:55:57.540 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48962.log.gz 2024-08-31T22:55:57.541 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72219.log.gz 2024-08-31T22:55:57.542 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72723.log.gz 2024-08-31T22:55:57.543 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67176.log 2024-08-31T22:55:57.543 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67827.log.gz 2024-08-31T22:55:57.544 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89799.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.88958.log.gz 2024-08-31T22:55:57.545 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67079.log.gz 2024-08-31T22:55:57.547 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.91238.log.gz 2024-08-31T22:55:57.548 INFO:teuthology.orchestra.run.smithi032.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.118165.log.gz 2024-08-31T22:55:57.550 INFO:teuthology.orchestra.run.smithi032.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118251.log.gz 2024-08-31T22:55:57.551 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58479.log 2024-08-31T22:55:57.554 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67176.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.115834.log.gz 2024-08-31T22:55:57.556 INFO:teuthology.orchestra.run.smithi032.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119768.log.gz 2024-08-31T22:55:57.557 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.93852.log.gz 2024-08-31T22:55:57.557 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58479.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50024.log 2024-08-31T22:55:57.557 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113112.log 2024-08-31T22:55:57.558 INFO:teuthology.orchestra.run.smithi032.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.107806.log.gz 2024-08-31T22:55:57.558 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30247.log.gz 2024-08-31T22:55:57.558 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34917.log.gz 2024-08-31T22:55:57.558 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.35072.log.gz 2024-08-31T22:55:57.559 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50024.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70350.log 2024-08-31T22:55:57.559 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.113112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161671.log 2024-08-31T22:55:57.560 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.70350.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61471.log 2024-08-31T22:55:57.561 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161671.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.161671.log.gz 2024-08-31T22:55:57.561 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38246.log 2024-08-31T22:55:57.562 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61471.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.162930.log 2024-08-31T22:55:57.563 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.38246.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60572.log 2024-08-31T22:55:57.564 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.162930.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.162930.log.gz 2024-08-31T22:55:57.566 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.142996.log.gz 2024-08-31T22:55:57.566 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135494.log 2024-08-31T22:55:57.567 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60572.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.85935.log.gz 2024-08-31T22:55:57.567 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111806.log 2024-08-31T22:55:57.568 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107748.log 2024-08-31T22:55:57.568 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83447.log.gz 2024-08-31T22:55:57.568 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.135494.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85001.log.gz 2024-08-31T22:55:57.569 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.111806.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97212.log.gz 2024-08-31T22:55:57.569 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.91477.log.gz 2024-08-31T22:55:57.570 INFO:teuthology.orchestra.run.smithi032.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.135494.log.gz 2024-08-31T22:55:57.570 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152333.log 2024-08-31T22:55:57.571 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107748.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157683.log 2024-08-31T22:55:57.573 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.152333.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.152333.log.gz 2024-08-31T22:55:57.573 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112120.log 2024-08-31T22:55:57.574 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157683.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54185.log 2024-08-31T22:55:57.575 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.112120.log: -- replaced with /var/log/ceph/cluster2-client.mirror.157683.log.gz 2024-08-31T22:55:57.575 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109151.log 2024-08-31T22:55:57.575 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.54185.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138121.log 2024-08-31T22:55:57.576 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.109151.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74494.log 2024-08-31T22:55:57.576 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.138121.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141052.log 2024-08-31T22:55:57.577 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.74494.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100913.log 2024-08-31T22:55:57.579 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.141052.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.141052.log.gz 2024-08-31T22:55:57.579 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118971.log 2024-08-31T22:55:57.580 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.100913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74273.log 2024-08-31T22:55:57.583 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118971.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85408.log 2024-08-31T22:55:57.583 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.74273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145314.log 2024-08-31T22:55:57.584 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.85408.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80831.log 2024-08-31T22:55:57.585 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.145314.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.145314.log.gz 2024-08-31T22:55:57.586 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119345.log 2024-08-31T22:55:57.588 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.80831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.163945.log 2024-08-31T22:55:57.588 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119345.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162661.log 2024-08-31T22:55:57.589 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.163945.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.163945.log.gz 2024-08-31T22:55:57.589 INFO:teuthology.orchestra.run.smithi032.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.139835.log.gz 2024-08-31T22:55:57.589 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123855.log 2024-08-31T22:55:57.591 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123421.log 2024-08-31T22:55:57.591 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.162661.log.gz 2024-08-31T22:55:57.591 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.123855.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123855.log.gz 2024-08-31T22:55:57.591 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144936.log 2024-08-31T22:55:57.592 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65211.log 2024-08-31T22:55:57.593 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.123421.log: /var/log/ceph/cluster1-client.mirror.144936.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123421.log.gz 2024-08-31T22:55:57.593 INFO:teuthology.orchestra.run.smithi032.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.123192.log.gz 2024-08-31T22:55:57.593 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144936.log.gz 2024-08-31T22:55:57.593 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83784.log 2024-08-31T22:55:57.593 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65211.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83633.log 2024-08-31T22:55:57.593 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83784.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38994.log 2024-08-31T22:55:57.594 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83633.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.54185.log.gz 2024-08-31T22:55:57.594 INFO:teuthology.orchestra.run.smithi032.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111080.log.gz 2024-08-31T22:55:57.594 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35444.log 2024-08-31T22:55:57.594 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69230.log.gz 2024-08-31T22:55:57.595 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37719.log 2024-08-31T22:55:57.595 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.38994.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114643.log 2024-08-31T22:55:57.595 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.35444.log: /var/log/ceph/cluster2-client.mirror.37719.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.60572.log.gz 2024-08-31T22:55:57.596 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60944.log.gz 2024-08-31T22:55:57.596 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74029.log 2024-08-31T22:55:57.596 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116020.log 2024-08-31T22:55:57.596 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.114643.log: /var/log/ceph/cluster2-client.mirror.74029.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51941.log.gz 2024-08-31T22:55:57.597 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162503.log 2024-08-31T22:55:57.597 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.116020.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.164210.log 2024-08-31T22:55:57.597 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39308.log 2024-08-31T22:55:57.598 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.79955.log.gz 2024-08-31T22:55:57.598 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.164210.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.101010.log.gz 2024-08-31T22:55:57.598 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162503.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37719.log.gz 2024-08-31T22:55:57.598 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104539.log.gz 2024-08-31T22:55:57.598 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37816.log.gz 2024-08-31T22:55:57.599 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44790.log.gz 2024-08-31T22:55:57.599 INFO:teuthology.orchestra.run.smithi032.stderr: 3.5% -- replaced with /var/log/ceph/cluster2-client.mirror.29369.log.gz 2024-08-31T22:55:57.599 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99142.log.gz 2024-08-31T22:55:57.599 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.164210.log.gz 2024-08-31T22:55:57.599 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.30097.log.gz 2024-08-31T22:55:57.599 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165889.log 2024-08-31T22:55:57.600 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39308.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77339.log 2024-08-31T22:55:57.601 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.admin.90636.log.gz 2024-08-31T22:55:57.601 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165889.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.94317.log.gz 2024-08-31T22:55:57.601 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38339.log.gz 2024-08-31T22:55:57.601 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.165889.log.gz 2024-08-31T22:55:57.601 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44604.log.gz 2024-08-31T22:55:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106252.log 2024-08-31T22:55:57.602 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.162503.log.gz/var/log/ceph/cluster2-client.mirror.77339.log: 2024-08-31T22:55:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48997.log 2024-08-31T22:55:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.106252.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72440.log 2024-08-31T22:55:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48997.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50082.log 2024-08-31T22:55:57.603 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70257.log 2024-08-31T22:55:57.603 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.89799.log.gz 2024-08-31T22:55:57.603 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50082.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161603.log 2024-08-31T22:55:57.604 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106159.log.gz 2024-08-31T22:55:57.604 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.105508.log.gz 2024-08-31T22:55:57.604 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.70257.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116152.log 2024-08-31T22:55:57.604 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161603.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.161603.log.gz 2024-08-31T22:55:57.605 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.72440.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46253.log 2024-08-31T22:55:57.606 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.116152.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50635.log 2024-08-31T22:55:57.606 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.46253.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37533.log 2024-08-31T22:55:57.606 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50635.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139449.log 2024-08-31T22:55:57.607 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.152375.log.gz 2024-08-31T22:55:57.607 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61599.log 2024-08-31T22:55:57.608 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.37533.log: /var/log/ceph/cluster2-client.mirror.139449.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153677.log 2024-08-31T22:55:57.608 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85652.log 2024-08-31T22:55:57.609 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61599.log: /var/log/ceph/cluster2-client.mirror.153677.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158178.log 2024-08-31T22:55:57.609 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.153677.log.gz 2024-08-31T22:55:57.609 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29335.log 2024-08-31T22:55:57.609 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.85652.log: /var/log/ceph/cluster2-client.mirror.158178.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46346.log 2024-08-31T22:55:57.610 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86307.log 2024-08-31T22:55:57.610 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.29335.log: /var/log/ceph/cluster2-client.mirror.46346.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109709.log 2024-08-31T22:55:57.610 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86307.log: 95.2% -- replaced with /var/log/ceph/cluster2-client.mirror.139449.log.gz 2024-08-31T22:55:57.611 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139350.log 2024-08-31T22:55:57.611 INFO:teuthology.orchestra.run.smithi032.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.158178.log.gz 2024-08-31T22:55:57.611 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.109709.log: /var/log/ceph/cluster2-client.mirror.84567.log 2024-08-31T22:55:57.612 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.139350.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141458.log 2024-08-31T22:55:57.612 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.84567.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.139350.log.gz 2024-08-31T22:55:57.612 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86811.log 2024-08-31T22:55:57.612 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.141458.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90357.log 2024-08-31T22:55:57.612 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86811.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141458.log.gz 2024-08-31T22:55:57.612 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81358.log 2024-08-31T22:55:57.613 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.90357.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54406.log 2024-08-31T22:55:57.613 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.81358.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36227.log 2024-08-31T22:55:57.613 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153366.log 2024-08-31T22:55:57.614 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.54406.log: /var/log/ceph/cluster2-client.mirror.36227.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157472.log 2024-08-31T22:55:57.614 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.153366.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58479.log.gz 2024-08-31T22:55:57.615 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150124.log 2024-08-31T22:55:57.617 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157472.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51193.log.gz 2024-08-31T22:55:57.617 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.153366.log.gz 2024-08-31T22:55:57.617 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161918.log 2024-08-31T22:55:57.617 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.157472.log.gz 2024-08-31T22:55:57.617 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.150124.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84850.log 2024-08-31T22:55:57.617 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9%/var/log/ceph/cluster1-client.mirror.161918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131994.log 2024-08-31T22:55:57.617 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.150124.log.gz 2024-08-31T22:55:57.617 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.161918.log.gz 2024-08-31T22:55:57.617 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.84850.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54464.log 2024-08-31T22:55:57.618 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37347.log 2024-08-31T22:55:57.618 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.131994.log: /var/log/ceph/cluster2-client.mirror.54464.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61750.log 2024-08-31T22:55:57.621 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62683.log 2024-08-31T22:55:57.621 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.37347.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131994.log.gz 2024-08-31T22:55:57.621 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61750.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96619.log 2024-08-31T22:55:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62683.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31146.log 2024-08-31T22:55:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96619.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42643.log 2024-08-31T22:55:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.31146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151477.log 2024-08-31T22:55:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.42643.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107000.log 2024-08-31T22:55:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.151477.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44945.log 2024-08-31T22:55:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107000.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86400.log 2024-08-31T22:55:57.622 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.151477.log.gz 2024-08-31T22:55:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.44945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29181.log 2024-08-31T22:55:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86400.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33576.log 2024-08-31T22:55:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.29181.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32022.log 2024-08-31T22:55:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33576.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145633.log 2024-08-31T22:55:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.32022.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157825.log 2024-08-31T22:55:57.623 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.145633.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153004.log 2024-08-31T22:55:57.624 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157825.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143754.log 2024-08-31T22:55:57.624 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.145633.log.gz 2024-08-31T22:55:57.624 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70257.log.gz 2024-08-31T22:55:57.624 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.157825.log.gz 2024-08-31T22:55:57.624 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140146.log 2024-08-31T22:55:57.624 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.153004.log: /var/log/ceph/cluster2-client.mirror.143754.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.143754.log.gz 2024-08-31T22:55:57.624 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35444.log.gz 2024-08-31T22:55:57.625 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160020.log 2024-08-31T22:55:57.625 INFO:teuthology.orchestra.run.smithi032.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.153004.log.gz 2024-08-31T22:55:57.626 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48528.log 2024-08-31T22:55:57.626 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.140146.log: /var/log/ceph/cluster2-client.mirror.160020.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43856.log 2024-08-31T22:55:57.626 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108647.log 2024-08-31T22:55:57.626 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.140146.log.gz 2024-08-31T22:55:57.626 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48528.log: /var/log/ceph/cluster2-client.mirror.43856.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83261.log 2024-08-31T22:55:57.627 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.108647.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152619.log 2024-08-31T22:55:57.627 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83261.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104818.log 2024-08-31T22:55:57.627 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.152619.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46943.log 2024-08-31T22:55:57.627 INFO:teuthology.orchestra.run.smithi032.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.160020.log.gz 2024-08-31T22:55:57.627 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.104818.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.152619.log.gz 2024-08-31T22:55:57.628 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66800.log 2024-08-31T22:55:57.629 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.46943.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156692.log 2024-08-31T22:55:57.629 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.66800.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58107.log 2024-08-31T22:55:57.629 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.163130.log 2024-08-31T22:55:57.629 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58107.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43170.log 2024-08-31T22:55:57.630 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81544.log.gz 2024-08-31T22:55:57.630 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.163130.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.163130.log.gz 2024-08-31T22:55:57.630 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.110236.log.gz 2024-08-31T22:55:57.631 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107841.log 2024-08-31T22:55:57.631 INFO:teuthology.orchestra.run.smithi032.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.156692.log.gz/var/log/ceph/cluster2-client.mirror.43170.log: 2024-08-31T22:55:57.631 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.78273.log.gz 2024-08-31T22:55:57.631 INFO:teuthology.orchestra.run.smithi032.stderr: 93.0%gzip -5 -- replaced with /var/log/ceph/cluster2-client.mirror.118008.log.gz --verbose 2024-08-31T22:55:57.631 INFO:teuthology.orchestra.run.smithi032.stderr: -- /var/log/ceph/cluster2-client.mirror.109488.log 2024-08-31T22:55:57.631 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107841.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156399.log 2024-08-31T22:55:57.632 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142187.log 2024-08-31T22:55:57.634 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.109488.log: /var/log/ceph/cluster2-client.mirror.156399.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.81017.log.gz 2024-08-31T22:55:57.634 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.142187.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.90229.log.gz 2024-08-31T22:55:57.634 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.142187.log.gz 2024-08-31T22:55:57.634 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34080.log.gz 2024-08-31T22:55:57.634 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.114643.log.gz 2024-08-31T22:55:57.634 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.84850.log.gz 2024-08-31T22:55:57.634 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76068.log.gz 2024-08-31T22:55:57.634 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102785.log 2024-08-31T22:55:57.634 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.83633.log.gz 2024-08-31T22:55:57.634 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.156399.log.gz 2024-08-31T22:55:57.634 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.155333.log 2024-08-31T22:55:57.634 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142862.log 2024-08-31T22:55:57.635 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.98580.log.gz 2024-08-31T22:55:57.636 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.102785.log: /var/log/ceph/cluster2-client.mirror.155333.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81637.log 2024-08-31T22:55:57.637 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.142862.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.155333.log.gz 2024-08-31T22:55:57.637 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.142862.log.gz 2024-08-31T22:55:57.637 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153411.log 2024-08-31T22:55:57.637 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.81637.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98146.log 2024-08-31T22:55:57.637 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44232.log 2024-08-31T22:55:57.637 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.153411.log: /var/log/ceph/cluster2-client.mirror.98146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84102.log 2024-08-31T22:55:57.637 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.44232.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103192.log 2024-08-31T22:55:57.637 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.111806.log.gz 2024-08-31T22:55:57.637 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.84102.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95720.log 2024-08-31T22:55:57.639 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.165265.log 2024-08-31T22:55:57.639 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.103192.log: /var/log/ceph/cluster2-client.mirror.95720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75371.log 2024-08-31T22:55:57.639 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.165265.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162818.log 2024-08-31T22:55:57.639 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57545.log.gz 2024-08-31T22:55:57.639 INFO:teuthology.orchestra.run.smithi032.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.153411.log.gz 2024-08-31T22:55:57.639 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.75371.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.165265.log.gz 2024-08-31T22:55:57.639 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28499.log 2024-08-31T22:55:57.639 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162818.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162458.log 2024-08-31T22:55:57.640 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.162818.log.gz 2024-08-31T22:55:57.640 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49427.log 2024-08-31T22:55:57.640 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28499.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28499.log.gz 2024-08-31T22:55:57.641 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162458.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109023.log 2024-08-31T22:55:57.641 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.162458.log.gz 2024-08-31T22:55:57.641 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49427.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.91049.log.gz 2024-08-31T22:55:57.641 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88117.log.gz 2024-08-31T22:55:57.641 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86063.log 2024-08-31T22:55:57.642 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.109023.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54406.log.gz 2024-08-31T22:55:57.642 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161176.log 2024-08-31T22:55:57.642 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.54964.log.gz 2024-08-31T22:55:57.642 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54464.log.gz 2024-08-31T22:55:57.642 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65339.log.gz 2024-08-31T22:55:57.643 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86063.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53437.log.gz 2024-08-31T22:55:57.643 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53937.log.gz 2024-08-31T22:55:57.644 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43914.log 88.1% 2024-08-31T22:55:57.644 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.61599.log.gz 2024-08-31T22:55:57.644 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61564.log.gz 2024-08-31T22:55:57.645 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72440.log.gz 2024-08-31T22:55:57.645 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161176.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72630.log.gz 2024-08-31T22:55:57.645 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.161176.log.gz 2024-08-31T22:55:57.645 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74401.log.gz 2024-08-31T22:55:57.645 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.155054.log.gz 2024-08-31T22:55:57.645 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.80831.log.gz 2024-08-31T22:55:57.645 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70385.log 88.6% 2024-08-31T22:55:57.645 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.74684.log.gz 2024-08-31T22:55:57.646 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102258.log.gz 2024-08-31T22:55:57.646 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118501.log 2024-08-31T22:55:57.646 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100727.log.gz 2024-08-31T22:55:57.646 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.102785.log.gz 2024-08-31T22:55:57.646 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103157.log.gz 2024-08-31T22:55:57.646 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.43914.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103192.log.gz 2024-08-31T22:55:57.646 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.70385.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103645.log.gz 2024-08-31T22:55:57.647 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99177.log 2024-08-31T22:55:57.647 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75975.log 2024-08-31T22:55:57.648 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118501.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.104818.log.gz 2024-08-31T22:55:57.648 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.99177.log: --verbose -- /var/log/ceph/cluster2-client.mirror.58572.log 2024-08-31T22:55:57.648 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75975.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37533.log.gz 2024-08-31T22:55:57.648 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118388.log 2024-08-31T22:55:57.648 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35479.log 2024-08-31T22:55:57.648 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58572.log: /var/log/ceph/cluster1-client.mirror.118388.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78963.log 2024-08-31T22:55:57.648 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.35479.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66521.log 2024-08-31T22:55:57.650 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/cluster2-client.mirror.78963.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55956.log 2024-08-31T22:55:57.650 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39308.log.gz 2024-08-31T22:55:57.650 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.66521.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66835.log 2024-08-31T22:55:57.650 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28674.log 2024-08-31T22:55:57.651 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.55956.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94375.log.gz 2024-08-31T22:55:57.651 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.66835.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45445.log 2024-08-31T22:55:57.651 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147396.log 2024-08-31T22:55:57.651 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.28674.log: /var/log/ceph/cluster2-client.mirror.45445.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62726.log 2024-08-31T22:55:57.651 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28674.log.gz 2024-08-31T22:55:57.651 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.147396.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73281.log 2024-08-31T22:55:57.651 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62726.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.147396.log.gz 2024-08-31T22:55:57.651 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65304.log 2024-08-31T22:55:57.654 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156934.log 2024-08-31T22:55:57.654 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.73281.log: /var/log/ceph/cluster2-client.mirror.65304.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163232.log 2024-08-31T22:55:57.654 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156934.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86249.log 2024-08-31T22:55:57.654 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163232.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74870.log 2024-08-31T22:55:57.654 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.163232.log.gz 2024-08-31T22:55:57.654 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86249.log: gzip -5 --verbose -- 89.3% /var/log/ceph/cluster2-client.mirror.159747.log 2024-08-31T22:55:57.654 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.156934.log.gz 2024-08-31T22:55:57.654 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.74870.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119907.log 2024-08-31T22:55:57.654 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151158.log 2024-08-31T22:55:57.654 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119907.log: /var/log/ceph/cluster2-client.mirror.159747.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36506.log 2024-08-31T22:55:57.654 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109244.log 2024-08-31T22:55:57.655 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.151158.log: /var/log/ceph/cluster2-client.mirror.36506.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119860.log 2024-08-31T22:55:57.655 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151158.log.gz 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.159747.log.gz 2024-08-31T22:55:57.655 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:57.655 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101138.log 2024-08-31T22:55:57.658 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141868.log 2024-08-31T22:55:57.658 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119860.log: /var/log/ceph/cluster2-client.mirror.101138.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67641.log 2024-08-31T22:55:57.658 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.141868.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101289.log 2024-08-31T22:55:57.658 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67641.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.141868.log.gz 2024-08-31T22:55:57.658 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38994.log.gz 2024-08-31T22:55:57.658 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.43170.log.gz 2024-08-31T22:55:57.658 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43856.log.gz 2024-08-31T22:55:57.658 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43914.log.gz 2024-08-31T22:55:57.658 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78463.log 2024-08-31T22:55:57.659 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48528.log.gz 2024-08-31T22:55:57.659 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101289.log: /var/log/ceph/cluster2-client.mirror.109244.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108740.log 2024-08-31T22:55:57.659 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.78463.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.165983.log 2024-08-31T22:55:57.659 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.108740.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141361.log 2024-08-31T22:55:57.659 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.165983.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46288.log 2024-08-31T22:55:57.659 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.165983.log.gz 2024-08-31T22:55:57.659 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.141361.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40862.log 2024-08-31T22:55:57.659 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.141361.log.gz 2024-08-31T22:55:57.659 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.46288.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.163871.log.gz 2024-08-31T22:55:57.664 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85749.log 2024-08-31T22:55:57.664 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.40862.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103378.log 2024-08-31T22:55:57.664 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49427.log.gz 2024-08-31T22:55:57.664 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.85749.log: --verbose -- /var/log/ceph/cluster2-client.mirror.96526.log 2024-08-31T22:55:57.664 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.103378.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51848.log 2024-08-31T22:55:57.665 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96526.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74556.log 2024-08-31T22:55:57.665 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51848.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140696.log 2024-08-31T22:55:57.665 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.74556.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115590.log 2024-08-31T22:55:57.665 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.140696.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66087.log 2024-08-31T22:55:57.665 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.115590.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28649.log 2024-08-31T22:55:57.665 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.66087.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115555.log 2024-08-31T22:55:57.665 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28649.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109395.log 2024-08-31T22:55:57.665 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28649.log.gz 2024-08-31T22:55:57.665 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.115555.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60758.log 2024-08-31T22:55:57.665 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.109395.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65773.log 2024-08-31T22:55:57.665 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60758.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146046.log 2024-08-31T22:55:57.665 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65773.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134772.log 2024-08-31T22:55:57.665 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.146046.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40804.log 2024-08-31T22:55:57.665 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.134772.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.146046.log.gz 2024-08-31T22:55:57.666 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36378.log 2024-08-31T22:55:57.666 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111269.log 2024-08-31T22:55:57.666 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.40804.log: /var/log/ceph/cluster2-client.mirror.36378.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60161.log 2024-08-31T22:55:57.666 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.111269.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48435.log 2024-08-31T22:55:57.666 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60161.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113674.log 2024-08-31T22:55:57.668 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51414.log 2024-08-31T22:55:57.669 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48435.log: /var/log/ceph/cluster2-client.mirror.113674.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160636.log 2024-08-31T22:55:57.669 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.93162.log.gz 2024-08-31T22:55:57.669 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51414.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95534.log 2024-08-31T22:55:57.669 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139793.log 2024-08-31T22:55:57.669 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.160636.log: /var/log/ceph/cluster2-client.mirror.95534.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.160636.log.gz 2024-08-31T22:55:57.669 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114360.log 2024-08-31T22:55:57.669 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.139793.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68110.log 2024-08-31T22:55:57.669 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.114360.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80273.log 2024-08-31T22:55:57.669 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.68110.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33111.log 2024-08-31T22:55:57.669 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.80273.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165327.log 2024-08-31T22:55:57.669 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29352.log 2024-08-31T22:55:57.669 INFO:teuthology.orchestra.run.smithi032.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.139793.log.gz 2024-08-31T22:55:57.669 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165327.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135514.log 2024-08-31T22:55:57.670 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.29352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87931.log 2024-08-31T22:55:57.670 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.135514.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69416.log 2024-08-31T22:55:57.670 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.87931.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.165327.log.gz 2024-08-31T22:55:57.670 INFO:teuthology.orchestra.run.smithi032.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.135514.log.gz 2024-08-31T22:55:57.671 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91701.log 2024-08-31T22:55:57.671 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.146731.log.gz 2024-08-31T22:55:57.671 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.69416.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40114.log 2024-08-31T22:55:57.672 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.82292.log.gz 2024-08-31T22:55:57.672 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.91701.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115741.log 2024-08-31T22:55:57.673 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.40114.log: 88.3%gzip 88.4% -5 --verbose -- replaced with /var/log/ceph/cluster2-client.mirror.82513.log.gz -- 2024-08-31T22:55:57.673 INFO:teuthology.orchestra.run.smithi032.stderr: /var/log/ceph/cluster2-client.mirror.69451.log 2024-08-31T22:55:57.673 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.82478.log.gz 2024-08-31T22:55:57.673 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52782.log.gz 2024-08-31T22:55:57.673 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107093.log 2024-08-31T22:55:57.673 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.69451.log: /var/log/ceph/cluster2-client.mirror.115741.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83819.log 2024-08-31T22:55:57.673 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70385.log.gz 2024-08-31T22:55:57.673 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107093.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111651.log 2024-08-31T22:55:57.673 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147087.log 2024-08-31T22:55:57.678 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83819.log: /var/log/ceph/cluster2-client.mirror.111651.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35479.log.gz 2024-08-31T22:55:57.678 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36378.log.gz 2024-08-31T22:55:57.678 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40114.log.gz 2024-08-31T22:55:57.678 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37781.log 2024-08-31T22:55:57.678 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.147087.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40804.log.gz 2024-08-31T22:55:57.678 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40862.log.gz 2024-08-31T22:55:57.678 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104105.log 2024-08-31T22:55:57.678 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.165868.log 2024-08-31T22:55:57.678 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.37781.log: /var/log/ceph/cluster2-client.mirror.104105.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21472.log 2024-08-31T22:55:57.679 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157379.log 2024-08-31T22:55:57.679 INFO:teuthology.orchestra.run.smithi032.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.147087.log.gz 2024-08-31T22:55:57.679 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph.tmp-client.admin.21472.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160142.log 2024-08-31T22:55:57.679 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21472.log.gz 2024-08-31T22:55:57.679 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157379.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36285.log 2024-08-31T22:55:57.679 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.160142.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33332.log 2024-08-31T22:55:57.679 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.36285.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29284.log 2024-08-31T22:55:57.679 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.165868.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.160142.log.gz 2024-08-31T22:55:57.679 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.157379.log.gz 2024-08-31T22:55:57.679 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163254.log 2024-08-31T22:55:57.679 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.165868.log.gz 2024-08-31T22:55:57.679 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33332.log: /var/log/ceph/cluster2-client.admin.29284.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78591.log 2024-08-31T22:55:57.679 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163254.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119365.log 2024-08-31T22:55:57.679 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.138121.log.gz 2024-08-31T22:55:57.680 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.163254.log.gz 2024-08-31T22:55:57.680 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.78591.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36192.log 2024-08-31T22:55:57.680 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147473.log 2024-08-31T22:55:57.688 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.36192.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42422.log 2024-08-31T22:55:57.688 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162166.log 2024-08-31T22:55:57.688 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.147473.log: /var/log/ceph/cluster2-client.mirror.42422.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87989.log 2024-08-31T22:55:57.688 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28599.log 2024-08-31T22:55:57.688 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162166.log: /var/log/ceph/cluster2-client.mirror.87989.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.162166.log.gz 2024-08-31T22:55:57.688 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135024.log 2024-08-31T22:55:57.688 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28599.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/cluster2-client.admin.28599.log.gz /var/log/ceph/cluster2-client.admin.25278.log 2024-08-31T22:55:57.688 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:57.688 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51007.log 2024-08-31T22:55:57.688 INFO:teuthology.orchestra.run.smithi032.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.147473.log.gz 2024-08-31T22:55:57.689 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.135024.log: /var/log/ceph/cluster2-client.admin.25278.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151203.log 2024-08-31T22:55:57.689 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25278.log.gz 2024-08-31T22:55:57.689 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135024.log.gz 2024-08-31T22:55:57.689 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51007.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82889.log 2024-08-31T22:55:57.689 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.151203.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85280.log 2024-08-31T22:55:57.689 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82889.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140171.log 2024-08-31T22:55:57.689 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.107841.log.gz 2024-08-31T22:55:57.689 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.85280.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62844.log 2024-08-31T22:55:57.689 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.140171.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119449.log 2024-08-31T22:55:57.689 INFO:teuthology.orchestra.run.smithi032.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151203.log.gz 2024-08-31T22:55:57.689 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.62844.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159332.log 2024-08-31T22:55:57.689 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112550.log 2024-08-31T22:55:57.689 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.119449.log: /var/log/ceph/cluster2-client.mirror.159332.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40897.log 2024-08-31T22:55:57.690 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.159332.log.gz 2024-08-31T22:55:57.690 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.112550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156721.log 2024-08-31T22:55:57.690 INFO:teuthology.orchestra.run.smithi032.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.140171.log.gz 2024-08-31T22:55:57.690 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161941.log 2024-08-31T22:55:57.690 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.40897.log: /var/log/ceph/cluster2-client.mirror.156721.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67955.log 2024-08-31T22:55:57.690 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161941.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63847.log 2024-08-31T22:55:57.690 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.156721.log.gz 2024-08-31T22:55:57.690 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.161941.log.gz 2024-08-31T22:55:57.690 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67955.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61971.log 2024-08-31T22:55:57.690 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63847.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160000.log 2024-08-31T22:55:57.690 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61971.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70665.log 2024-08-31T22:55:57.690 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.160000.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45916.log 2024-08-31T22:55:57.690 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.70665.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.160000.log.gz 2024-08-31T22:55:57.690 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109899.log 2024-08-31T22:55:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.45916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82327.log 2024-08-31T22:55:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.109899.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90863.log 2024-08-31T22:55:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82327.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97739.log 2024-08-31T22:55:57.691 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109023.log.gz 2024-08-31T22:55:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88396.log 2024-08-31T22:55:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.90863.log: /var/log/ceph/cluster2-client.mirror.97739.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65494.log 2024-08-31T22:55:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.88396.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158391.log 2024-08-31T22:55:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65494.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.111651.log.gz 2024-08-31T22:55:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158391.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.78963.log.gz 2024-08-31T22:55:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156328.log 2024-08-31T22:55:57.730 INFO:teuthology.orchestra.run.smithi007.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-08-31T22:55:57.736 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67362.log 2024-08-31T22:55:57.736 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.113674.log.gz 2024-08-31T22:55:57.736 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30212.log.gz 2024-08-31T22:55:57.736 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156328.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32305.log 2024-08-31T22:55:57.736 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76816.log.gz 2024-08-31T22:55:57.736 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67362.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119017.log 2024-08-31T22:55:57.736 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.156328.log.gz 2024-08-31T22:55:57.737 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.32305.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143360.log 2024-08-31T22:55:57.737 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29454.log 2024-08-31T22:55:57.737 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119017.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.158391.log.gz 2024-08-31T22:55:57.737 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.143360.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165080.log 2024-08-31T22:55:57.737 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.29454.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95627.log 2024-08-31T22:55:57.737 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29454.log.gz 2024-08-31T22:55:57.737 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165080.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89985.log 2024-08-31T22:55:57.737 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.95627.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33704.log 2024-08-31T22:55:57.737 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86063.log.gz 2024-08-31T22:55:57.737 INFO:teuthology.orchestra.run.smithi032.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.143360.log.gz 2024-08-31T22:55:57.737 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89985.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30530.log 2024-08-31T22:55:57.737 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33704.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105039.log 2024-08-31T22:55:57.737 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.30530.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32735.log 2024-08-31T22:55:57.738 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105039.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82889.log.gz 2024-08-31T22:55:57.738 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.91701.log.gz 2024-08-31T22:55:57.738 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141836.log 2024-08-31T22:55:57.738 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.95534.log.gz 2024-08-31T22:55:57.738 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141032.log 2024-08-31T22:55:57.739 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.141836.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53902.log 2024-08-31T22:55:57.739 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.141032.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141836.log.gz 2024-08-31T22:55:57.740 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.32735.log: /var/log/ceph/cluster2-client.mirror.53902.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.95627.log.gz 2024-08-31T22:55:57.740 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165651.log 2024-08-31T22:55:57.740 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.95720.log.gzgzip 2024-08-31T22:55:57.740 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157874.log 2024-08-31T22:55:57.740 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165651.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87334.log 2024-08-31T22:55:57.740 INFO:teuthology.orchestra.run.smithi032.stderr: 83.6% -- replaced with /var/log/ceph/cluster2-client.mirror.165651.log.gz 2024-08-31T22:55:57.740 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157874.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151126.log 2024-08-31T22:55:57.740 INFO:teuthology.orchestra.run.smithi032.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.141032.log.gz 2024-08-31T22:55:57.740 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3%/var/log/ceph/cluster2-client.mirror.87334.log: -- replaced with /var/log/ceph/cluster2-client.mirror.157874.log.gz 2024-08-31T22:55:57.740 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116803.log 2024-08-31T22:55:57.740 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.151126.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80738.log 2024-08-31T22:55:57.740 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.151126.log.gz 2024-08-31T22:55:57.740 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.116803.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93348.log 2024-08-31T22:55:57.740 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.80738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154646.log 2024-08-31T22:55:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.93348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31650.log 2024-08-31T22:55:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.154646.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54123.log 2024-08-31T22:55:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.31650.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68517.log 2024-08-31T22:55:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.54123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117551.log 2024-08-31T22:55:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.68517.log: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.154646.log.gz 2024-08-31T22:55:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34882.log 2024-08-31T22:55:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.117551.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91299.log 2024-08-31T22:55:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.34882.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85036.log 2024-08-31T22:55:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.91299.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135469.log 2024-08-31T22:55:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.85036.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161873.log 2024-08-31T22:55:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.135469.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31278.log 2024-08-31T22:55:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161873.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135469.log.gz 2024-08-31T22:55:57.742 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115869.log 2024-08-31T22:55:57.742 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.161873.log.gz/var/log/ceph/cluster2-client.mirror.31278.log: 2024-08-31T22:55:57.742 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110608.log 2024-08-31T22:55:57.742 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.115869.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31522.log 2024-08-31T22:55:57.742 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/cluster2-client.mirror.110608.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.164000.log 2024-08-31T22:55:57.742 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.31522.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55526.log 2024-08-31T22:55:57.742 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/cluster1-client.mirror.164000.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116582.log 2024-08-31T22:55:57.742 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.55526.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.164000.log.gz 2024-08-31T22:55:57.742 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119837.log 2024-08-31T22:55:57.742 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.116582.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35293.log 2024-08-31T22:55:57.742 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119837.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112364.log 2024-08-31T22:55:57.742 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.35293.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91411.log 2024-08-31T22:55:57.742 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.112364.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151989.log 2024-08-31T22:55:57.743 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.91411.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102444.log 2024-08-31T22:55:57.743 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.151989.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116396.log 2024-08-31T22:55:57.743 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.102444.log: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.140696.log.gz 2024-08-31T22:55:57.743 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119627.log 2024-08-31T22:55:57.743 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/cluster2-client.mirror.116396.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118548.log 2024-08-31T22:55:57.743 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119627.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51604.log 2024-08-31T22:55:57.743 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118548.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77374.log 2024-08-31T22:55:57.743 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51604.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49462.log 2024-08-31T22:55:57.743 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77374.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116768.log 2024-08-31T22:55:57.743 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49462.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147873.log 2024-08-31T22:55:57.743 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.116768.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157492.log 2024-08-31T22:55:57.743 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.147873.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72347.log 2024-08-31T22:55:57.743 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157492.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92883.log 2024-08-31T22:55:57.744 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.72347.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163668.log 2024-08-31T22:55:57.744 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.92883.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60475.log 2024-08-31T22:55:57.744 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163668.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149745.log 2024-08-31T22:55:57.744 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60475.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.157492.log.gz 2024-08-31T22:55:57.744 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115213.log 2024-08-31T22:55:57.744 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.149745.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.163668.log.gz 2024-08-31T22:55:57.744 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34452.log 2024-08-31T22:55:57.744 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.115213.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75471.log 2024-08-31T22:55:57.744 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.34452.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34545.log 2024-08-31T22:55:57.744 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75471.log: gzip -5 --verbose -- 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.149745.log.gz 2024-08-31T22:55:57.744 INFO:teuthology.orchestra.run.smithi032.stderr: /var/log/ceph/cluster2-client.mirror.48028.log 2024-08-31T22:55:57.744 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.34545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.163616.log 96.5% -- replaced with /var/log/ceph/cluster2-client.mirror.147873.log.gz 2024-08-31T22:55:57.745 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:57.745 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48028.log: gzip 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57673.log.gz 2024-08-31T22:55:57.745 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65494.log.gz 2024-08-31T22:55:57.745 INFO:teuthology.orchestra.run.smithi032.stderr: /var/log/ceph/cluster2-client.mirror.111003.log 2024-08-31T22:55:57.745 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.163616.log: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72498.log 2024-08-31T22:55:57.745 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69416.log.gz 2024-08-31T22:55:57.745 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.163616.log.gz 2024-08-31T22:55:57.745 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69451.log.gz 2024-08-31T22:55:57.745 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.111003.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.118341.log.gz 2024-08-31T22:55:57.745 INFO:teuthology.orchestra.run.smithi032.stderr:gzip 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.51007.log.gz 2024-08-31T22:55:57.745 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38808.log/var/log/ceph/cluster2-client.mirror.72498.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53902.log.gz 2024-08-31T22:55:57.745 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60475.log.gz 2024-08-31T22:55:57.746 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62405.log.gz 2024-08-31T22:55:57.746 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77843.log 2024-08-31T22:55:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.38808.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66145.log 2024-08-31T22:55:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96371.log 2024-08-31T22:55:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77843.log: /var/log/ceph/cluster2-client.mirror.66145.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64874.log 2024-08-31T22:55:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150066.log 2024-08-31T22:55:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64874.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160466.log 2024-08-31T22:55:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.150066.log: gzip 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.150066.log.gz 2024-08-31T22:55:57.746 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44290.log 2024-08-31T22:55:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.160466.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147495.log 2024-08-31T22:55:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.44290.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136458.log 2024-08-31T22:55:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.147495.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55398.log 2024-08-31T22:55:57.747 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.136458.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73153.log.gz 2024-08-31T22:55:57.747 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.55398.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103564.log.gz 2024-08-31T22:55:57.747 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52069.log 2024-08-31T22:55:57.747 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.52069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55770.log 2024-08-31T22:55:57.747 INFO:teuthology.orchestra.run.smithi032.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.136458.log.gz 2024-08-31T22:55:57.747 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140562.log 2024-08-31T22:55:57.747 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.55770.log: /var/log/ceph/cluster2-client.mirror.140562.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.160466.log.gz 2024-08-31T22:55:57.747 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59696.log 2024-08-31T22:55:57.747 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.140562.log.gz 2024-08-31T22:55:57.747 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99177.log.gz 2024-08-31T22:55:57.747 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105039.log.gz 2024-08-31T22:55:57.747 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.59696.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96561.log 2024-08-31T22:55:57.747 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.111003.log.gz 2024-08-31T22:55:57.748 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.112550.log.gz 2024-08-31T22:55:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96561.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87803.log 2024-08-31T22:55:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.87803.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.115741.log.gz 2024-08-31T22:55:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88923.log 2024-08-31T22:55:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:gzip 88.3% -5 -- replaced with /var/log/ceph/cluster2-client.mirror.59696.log.gz --verbose 2024-08-31T22:55:57.748 INFO:teuthology.orchestra.run.smithi032.stderr: -- /var/log/ceph/cluster2-client.mirror.95813.log 2024-08-31T22:55:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.88923.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33890.log 2024-08-31T22:55:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.95813.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166397.log 2024-08-31T22:55:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33890.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165054.log 2024-08-31T22:55:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.166397.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166397.log.gz 2024-08-31T22:55:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156885.log 2024-08-31T22:55:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165054.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34789.log 2024-08-31T22:55:57.749 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.165054.log.gz 2024-08-31T22:55:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156885.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82420.log 2024-08-31T22:55:57.749 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.156885.log.gz 2024-08-31T22:55:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.34789.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.83819.log.gz 2024-08-31T22:55:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106907.log 2024-08-31T22:55:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.82420.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105473.log.gz 2024-08-31T22:55:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47780.log 2024-08-31T22:55:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.106907.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121110.log 2024-08-31T22:55:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.47780.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43949.log 2024-08-31T22:55:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60909.log 2024-08-31T22:55:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.121110.log: /var/log/ceph/cluster2-client.mirror.43949.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.121110.log.gz 2024-08-31T22:55:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60909.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144617.log 2024-08-31T22:55:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93069.log 2024-08-31T22:55:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.144617.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68703.log 2024-08-31T22:55:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.93069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39870.log 2024-08-31T22:55:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.68703.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118861.log 2024-08-31T22:55:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39870.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58421.log 2024-08-31T22:55:57.750 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37347.log.gz 2024-08-31T22:55:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118861.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75696.log 2024-08-31T22:55:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58421.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.154441.log 2024-08-31T22:55:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75696.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64839.log 2024-08-31T22:55:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144105.log 2024-08-31T22:55:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.154441.log: /var/log/ceph/cluster2-client.mirror.64839.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.154441.log.gz 2024-08-31T22:55:57.755 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28473.log 2024-08-31T22:55:57.755 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.144105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63998.log 2024-08-31T22:55:57.755 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.144105.log.gz 2024-08-31T22:55:57.755 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28473.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28473.log.gz 2024-08-31T22:55:57.755 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48028.log.gz 2024-08-31T22:55:57.755 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55677.log 2024-08-31T22:55:57.755 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51604.log.gz 2024-08-31T22:55:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63998.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68703.log.gz 2024-08-31T22:55:57.756 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82420.log.gz 2024-08-31T22:55:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118950.log 2024-08-31T22:55:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.55677.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123169.log 2024-08-31T22:55:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118950.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121427.log 2024-08-31T22:55:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.123169.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160883.log 2024-08-31T22:55:57.756 INFO:teuthology.orchestra.run.smithi032.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.123169.log.gz 2024-08-31T22:55:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.121427.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102037.log 2024-08-31T22:55:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103722.log 2024-08-31T22:55:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.160883.log: /var/log/ceph/cluster2-client.mirror.102037.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.160883.log.gz 2024-08-31T22:55:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44046.log 2024-08-31T22:55:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.103722.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74649.log 2024-08-31T22:55:57.757 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.44046.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39835.log 2024-08-31T22:55:57.757 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.74649.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35351.log 2024-08-31T22:55:57.757 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39835.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139427.log 2024-08-31T22:55:57.757 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67176.log.gz 2024-08-31T22:55:57.757 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.35351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.163643.log 2024-08-31T22:55:57.757 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.139427.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165501.log 2024-08-31T22:55:57.757 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.163643.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.163643.log.gz 2024-08-31T22:55:57.757 INFO:teuthology.orchestra.run.smithi032.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.139427.log.gz 2024-08-31T22:55:57.757 INFO:teuthology.orchestra.run.smithi032.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118929.log.gz 2024-08-31T22:55:57.757 INFO:teuthology.orchestra.run.smithi032.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118971.log.gz 2024-08-31T22:55:57.757 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55526.log.gz 2024-08-31T22:55:57.757 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50082.log.gz 2024-08-31T22:55:57.758 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165501.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138483.log 2024-08-31T22:55:57.758 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60161.log.gz 2024-08-31T22:55:57.758 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60196.log.gz 2024-08-31T22:55:57.758 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113240.log 2024-08-31T22:55:57.760 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.138483.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138483.log.gz 2024-08-31T22:55:57.760 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66180.log 2024-08-31T22:55:57.760 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.113240.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106779.log 2024-08-31T22:55:57.760 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32305.log.gz 2024-08-31T22:55:57.760 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.66180.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36285.log.gz 2024-08-31T22:55:57.760 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46160.log 2024-08-31T22:55:57.760 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40300.log.gz 2024-08-31T22:55:57.760 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.106779.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117051.log 2024-08-31T22:55:57.760 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.46160.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64467.log 2024-08-31T22:55:57.761 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.117051.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.155426.log 2024-08-31T22:55:57.761 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64467.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66273.log 2024-08-31T22:55:57.762 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.155426.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.155426.log.gz 2024-08-31T22:55:57.762 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106349.log 2024-08-31T22:55:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.66273.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36599.log 2024-08-31T22:55:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.106349.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83881.log 2024-08-31T22:55:57.764 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.36599.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76967.log 2024-08-31T22:55:57.765 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83881.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75564.log.gz 2024-08-31T22:55:57.767 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89764.log 2024-08-31T22:55:57.767 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.76967.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28057.log 2024-08-31T22:55:57.768 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89764.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159463.log 2024-08-31T22:55:57.769 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28057.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.109899.log.gz 2024-08-31T22:55:57.769 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28057.log.gz 2024-08-31T22:55:57.770 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.78928.log.gz 2024-08-31T22:55:57.770 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.89985.log.gz 2024-08-31T22:55:57.770 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.113709.log.gz 2024-08-31T22:55:57.775 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159139.log 2024-08-31T22:55:57.776 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159463.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.159463.log.gz 2024-08-31T22:55:57.776 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44232.log.gz 2024-08-31T22:55:57.777 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117458.log 2024-08-31T22:55:57.777 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34789.log.gz 2024-08-31T22:55:57.778 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.159139.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35351.log.gz 2024-08-31T22:55:57.779 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67362.log.gz 2024-08-31T22:55:57.779 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87334.log.gz 2024-08-31T22:55:57.779 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.87803.log.gz 2024-08-31T22:55:57.780 INFO:teuthology.orchestra.run.smithi032.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.159139.log.gz 2024-08-31T22:55:57.781 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28849.log 2024-08-31T22:55:57.781 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.92918.log.gz 2024-08-31T22:55:57.782 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.117458.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28221.log 2024-08-31T22:55:57.782 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28849.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28849.log.gz 2024-08-31T22:55:57.782 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99456.log 2024-08-31T22:55:57.783 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28221.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28221.log.gz 2024-08-31T22:55:57.783 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81079.log 2024-08-31T22:55:57.784 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.99456.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69920.log 2024-08-31T22:55:57.785 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.81079.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61843.log 2024-08-31T22:55:57.788 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.69920.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135847.log 2024-08-31T22:55:57.788 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61843.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161986.log 2024-08-31T22:55:57.789 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.135847.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135847.log.gz 2024-08-31T22:55:57.790 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94600.log 2024-08-31T22:55:57.791 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161986.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.161986.log.gz 2024-08-31T22:55:57.791 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44139.log 2024-08-31T22:55:57.792 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94600.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127792.log 2024-08-31T22:55:57.793 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.44139.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86528.log.gz 2024-08-31T22:55:57.794 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83726.log.gz 2024-08-31T22:55:57.794 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.117086.log.gz 2024-08-31T22:55:57.794 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97681.log 2024-08-31T22:55:57.795 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.127792.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.127792.log.gz 2024-08-31T22:55:57.795 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119565.log 2024-08-31T22:55:57.801 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.97681.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49148.log 2024-08-31T22:55:57.802 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119565.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54650.log 2024-08-31T22:55:57.803 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.49148.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162728.log 2024-08-31T22:55:57.804 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.54650.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149461.log 2024-08-31T22:55:57.805 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162728.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.162728.log.gz 2024-08-31T22:55:57.805 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38750.log 2024-08-31T22:55:57.806 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.149461.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70478.log.gz 2024-08-31T22:55:57.806 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70665.log.gz 2024-08-31T22:55:57.807 INFO:teuthology.orchestra.run.smithi032.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.149461.log.gz 2024-08-31T22:55:57.808 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139382.log 2024-08-31T22:55:57.808 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.77843.log.gz 2024-08-31T22:55:57.809 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.38750.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66145.log.gz 2024-08-31T22:55:57.809 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66180.log.gz 2024-08-31T22:55:57.810 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165610.log 2024-08-31T22:55:57.810 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62626.log.gz 2024-08-31T22:55:57.811 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.139382.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139382.log.gz 2024-08-31T22:55:57.811 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142284.log 2024-08-31T22:55:57.812 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165610.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.165610.log.gz 2024-08-31T22:55:57.815 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92449.log 2024-08-31T22:55:57.815 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.142284.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.118521.log.gz 2024-08-31T22:55:57.815 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63998.log.gz 2024-08-31T22:55:57.816 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.142284.log.gz 2024-08-31T22:55:57.816 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44569.log 2024-08-31T22:55:57.816 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38808.log.gz 2024-08-31T22:55:57.817 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72498.log.gz 2024-08-31T22:55:57.817 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.92449.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73339.log.gz 2024-08-31T22:55:57.817 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.101661.log.gz 2024-08-31T22:55:57.817 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.101886.log.gz 2024-08-31T22:55:57.818 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.103064.log.gz 2024-08-31T22:55:57.818 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106779.log.gz 2024-08-31T22:55:57.819 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.83784.log.gz 2024-08-31T22:55:57.819 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.96561.log.gz 2024-08-31T22:55:57.819 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83881.log.gz 2024-08-31T22:55:57.819 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50635.log.gz 2024-08-31T22:55:57.820 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50856.log 2024-08-31T22:55:57.820 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.44569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37684.log 2024-08-31T22:55:57.820 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50856.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55677.log.gz 2024-08-31T22:55:57.821 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113112.log.gz 2024-08-31T22:55:57.822 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31650.log.gz 2024-08-31T22:55:57.823 INFO:teuthology.orchestra.run.smithi032.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118950.log.gz 2024-08-31T22:55:57.824 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35948.log 2024-08-31T22:55:57.825 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.37684.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118728.log.gz 2024-08-31T22:55:57.825 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158229.log 2024-08-31T22:55:57.826 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.35948.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.27987.log 2024-08-31T22:55:57.827 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158229.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.158229.log.gz 2024-08-31T22:55:57.827 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85784.log 2024-08-31T22:55:57.828 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.27987.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.27987.log.gz 2024-08-31T22:55:57.828 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129275.log 2024-08-31T22:55:57.829 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.85784.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142897.log 2024-08-31T22:55:57.835 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.129275.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.85652.log.gz 2024-08-31T22:55:57.838 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46288.log.gz 2024-08-31T22:55:57.839 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69920.log.gz 2024-08-31T22:55:57.840 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.77339.log.gz 2024-08-31T22:55:57.840 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35948.log.gz 2024-08-31T22:55:57.840 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.36006.log.gz 2024-08-31T22:55:57.840 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36227.log.gz 2024-08-31T22:55:57.840 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.36506.log.gz 2024-08-31T22:55:57.841 INFO:teuthology.orchestra.run.smithi032.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118501.log.gz 2024-08-31T22:55:57.841 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38750.log.gz 2024-08-31T22:55:57.842 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.109395.log.gz 2024-08-31T22:55:57.847 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81300.log.gz 2024-08-31T22:55:57.849 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.90357.log.gz 2024-08-31T22:55:57.851 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161401.log 2024-08-31T22:55:57.851 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34231.log.gz 2024-08-31T22:55:57.851 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.142897.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77095.log.gz 2024-08-31T22:55:57.852 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85784.log.gz 2024-08-31T22:55:57.852 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.142897.log.gz 2024-08-31T22:55:57.852 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86400.log.gz 2024-08-31T22:55:57.852 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.85749.log.gz 2024-08-31T22:55:57.852 INFO:teuthology.orchestra.run.smithi032.stderr: 95.2% -- replaced with /var/log/ceph/cluster2-client.mirror.165501.log.gz 2024-08-31T22:55:57.852 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89330.log 2024-08-31T22:55:57.853 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161401.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.161401.log.gz 2024-08-31T22:55:57.854 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41802.log 2024-08-31T22:55:57.855 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89330.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79246.log 2024-08-31T22:55:57.856 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.41802.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56890.log 2024-08-31T22:55:57.856 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.79246.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73843.log 2024-08-31T22:55:57.857 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.56890.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96747.log 2024-08-31T22:55:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.73843.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.92449.log.gz 2024-08-31T22:55:57.858 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.92976.log.gz 2024-08-31T22:55:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57921.log 2024-08-31T22:55:57.859 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96747.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116861.log 2024-08-31T22:55:57.860 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.57921.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145043.log 2024-08-31T22:55:57.861 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.116861.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28874.log 2024-08-31T22:55:57.863 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.145043.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.145043.log.gz 2024-08-31T22:55:57.863 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66707.log 2024-08-31T22:55:57.864 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28874.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28874.log.gz 2024-08-31T22:55:57.864 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163823.log 2024-08-31T22:55:57.865 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.66707.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56274.log 2024-08-31T22:55:57.866 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163823.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.163823.log.gz 2024-08-31T22:55:57.866 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75302.log 2024-08-31T22:55:57.867 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.56274.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28524.log 2024-08-31T22:55:57.868 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75302.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51976.log 2024-08-31T22:55:57.869 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.28524.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28524.log.gz 2024-08-31T22:55:57.869 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73246.log 2024-08-31T22:55:57.870 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51976.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39273.log 2024-08-31T22:55:57.871 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.73246.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83412.log 2024-08-31T22:55:57.871 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39273.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29478.log 2024-08-31T22:55:57.872 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83412.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38467.log 2024-08-31T22:55:57.873 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.29478.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29478.log.gz 2024-08-31T22:55:57.879 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45259.log 2024-08-31T22:55:57.880 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.38467.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38374.log 2024-08-31T22:55:57.881 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.45259.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72312.log 2024-08-31T22:55:57.881 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.38374.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64370.log 2024-08-31T22:55:57.883 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.72312.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56425.log 2024-08-31T22:55:57.884 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64370.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141490.log 2024-08-31T22:55:57.885 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.56425.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30960.log 2024-08-31T22:55:57.886 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.141490.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.141490.log.gz 2024-08-31T22:55:57.886 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143786.log 2024-08-31T22:55:57.889 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.30960.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120032.log 2024-08-31T22:55:57.889 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.143786.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.143786.log.gz 2024-08-31T22:55:57.889 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100948.log 2024-08-31T22:55:57.889 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.120032.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.144617.log.gz 2024-08-31T22:55:57.889 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33518.log 2024-08-31T22:55:57.890 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.100948.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130877.log 2024-08-31T22:55:57.890 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33518.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111899.log 2024-08-31T22:55:57.891 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100448.log 2024-08-31T22:55:57.891 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.130877.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.130877.log.gz 2024-08-31T22:55:57.892 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.111899.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.164542.log 2024-08-31T22:55:57.893 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.100448.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148259.log 2024-08-31T22:55:57.893 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48811.log 2024-08-31T22:55:57.894 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.164542.log: /var/log/ceph/cluster2-client.mirror.148259.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.164542.log.gz 2024-08-31T22:55:57.894 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141913.log 2024-08-31T22:55:57.895 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48811.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70013.log 2024-08-31T22:55:57.897 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.141913.log: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.141913.log.gz 2024-08-31T22:55:57.898 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119389.log 2024-08-31T22:55:57.899 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.70013.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69137.log 2024-08-31T22:55:57.900 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.119389.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148538.log 2024-08-31T22:55:57.901 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.69137.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29113.log 2024-08-31T22:55:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.148538.log: 88.0% 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54650.log.gz 2024-08-31T22:55:57.902 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.148538.log.gz 2024-08-31T22:55:57.902 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66707.log.gz 2024-08-31T22:55:57.903 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66800.log.gz 2024-08-31T22:55:57.903 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69137.log.gz 2024-08-31T22:55:57.903 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61750.log.gz 2024-08-31T22:55:57.904 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72312.log.gz 2024-08-31T22:55:57.904 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53247.log.gz 2024-08-31T22:55:57.904 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66835.log.gz 2024-08-31T22:55:57.904 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74870.log.gz 2024-08-31T22:55:57.904 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100948.log.gz 2024-08-31T22:55:57.904 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73281.log.gz 2024-08-31T22:55:57.904 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100913.log.gz 2024-08-31T22:55:57.905 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37684.log.gz 2024-08-31T22:55:57.905 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106252.log.gz 2024-08-31T22:55:57.905 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73246.log.gz 2024-08-31T22:55:57.905 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44569.log.gz 2024-08-31T22:55:57.905 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73843.log.gz 2024-08-31T22:55:57.905 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32828.log 2024-08-31T22:55:57.906 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.29113.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103529.log 2024-08-31T22:55:57.906 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.32828.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96057.log 2024-08-31T22:55:57.907 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.103529.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85842.log 2024-08-31T22:55:57.908 INFO:teuthology.orchestra.run.smithi032.stderr: 57.1% -- replaced with /var/log/ceph/cluster2-client.mirror.29335.log.gz 2024-08-31T22:55:57.908 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96057.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50856.log.gz 2024-08-31T22:55:57.908 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84102.log.gz 2024-08-31T22:55:57.908 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160424.log 2024-08-31T22:55:57.908 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46160.log.gz 2024-08-31T22:55:57.909 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.85842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161806.log 2024-08-31T22:55:57.909 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.89330.log.gz 2024-08-31T22:55:57.910 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.160424.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.160424.log.gz 2024-08-31T22:55:57.911 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37126.log 2024-08-31T22:55:57.911 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161806.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.161806.log.gz 2024-08-31T22:55:57.912 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58890.log 2024-08-31T22:55:57.912 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.37126.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.163508.log 2024-08-31T22:55:57.914 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58890.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.119277.log.gz 2024-08-31T22:55:57.914 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70164.log 2024-08-31T22:55:57.915 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163508.log: 83.6% -- replaced with /var/log/ceph/cluster2-client.mirror.163508.log.gz 2024-08-31T22:55:57.915 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112399.log 2024-08-31T22:55:57.916 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.70164.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.120092.log.gz 2024-08-31T22:55:57.919 INFO:teuthology.orchestra.run.smithi032.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119017.log.gz 2024-08-31T22:55:57.919 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149302.log 2024-08-31T22:55:57.920 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.112399.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58107.log.gz 2024-08-31T22:55:57.920 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144904.log 2024-08-31T22:55:57.921 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.149302.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.113240.log.gz 2024-08-31T22:55:57.921 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.149302.log.gz 2024-08-31T22:55:57.922 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.113581.log.gz 2024-08-31T22:55:57.923 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68517.log.gz 2024-08-31T22:55:57.923 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50821.log 2024-08-31T22:55:57.924 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.144904.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41802.log.gz 2024-08-31T22:55:57.924 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.144904.log.gz 2024-08-31T22:55:57.924 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36750.log 2024-08-31T22:55:57.925 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50821.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62250.log 2024-08-31T22:55:57.926 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.36750.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56890.log.gz 2024-08-31T22:55:57.929 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70164.log.gz 2024-08-31T22:55:57.929 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44945.log.gz 2024-08-31T22:55:57.929 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.29589.log 2024-08-31T22:55:57.929 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108740.log.gz 2024-08-31T22:55:57.930 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62250.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.109151.log.gz 2024-08-31T22:55:57.930 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160162.log 2024-08-31T22:55:57.931 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.110364.log.gz 2024-08-31T22:55:57.931 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.1.29589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57115.log 2024-08-31T22:55:57.933 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.160162.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.160162.log.gz 2024-08-31T22:55:57.934 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.81823.log.gz 2024-08-31T22:55:57.935 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160517.log 2024-08-31T22:55:57.935 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.57115.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118838.log 2024-08-31T22:55:57.936 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.160517.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.160517.log.gz 2024-08-31T22:55:57.937 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61657.log 2024-08-31T22:55:57.938 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.118838.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33576.log.gz 2024-08-31T22:55:57.938 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33518.log.gz 2024-08-31T22:55:57.938 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61657.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141535.log 2024-08-31T22:55:57.938 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.92414.log.gz 2024-08-31T22:55:57.940 INFO:teuthology.orchestra.run.smithi032.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121427.log.gz 2024-08-31T22:55:57.940 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56739.log 2024-08-31T22:55:57.941 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41233.log 2024-08-31T22:55:57.942 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.141535.log: /var/log/ceph/cluster2-client.mirror.56739.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.110740.log.gz 2024-08-31T22:55:57.943 INFO:teuthology.orchestra.run.smithi032.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141535.log.gz 2024-08-31T22:55:57.943 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.163107.log 2024-08-31T22:55:57.944 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117982.log 2024-08-31T22:55:57.944 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.41233.log: /var/log/ceph/cluster1-client.mirror.163107.log: 86.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128343.log 2024-08-31T22:55:57.945 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.163107.log.gz 2024-08-31T22:55:57.945 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.117982.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77467.log 2024-08-31T22:55:57.945 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.128343.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66370.log 2024-08-31T22:55:57.946 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77467.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.91949.log.gz 2024-08-31T22:55:57.946 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.98146.log.gz 2024-08-31T22:55:57.950 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39835.log.gz 87.9% 2024-08-31T22:55:57.950 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.39870.log.gz 2024-08-31T22:55:57.951 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79583.log 2024-08-31T22:55:57.952 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.66370.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140654.log 2024-08-31T22:55:57.952 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.79583.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48586.log 2024-08-31T22:55:57.955 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.140654.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.140654.log.gz 2024-08-31T22:55:57.955 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147109.log 2024-08-31T22:55:57.956 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48586.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137787.log 2024-08-31T22:55:57.961 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.147109.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.111899.log.gz 2024-08-31T22:55:57.963 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.112399.log.gz 2024-08-31T22:55:57.967 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93534.log 2024-08-31T22:55:57.967 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45445.log.gz 2024-08-31T22:55:57.967 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66087.log.gz 2024-08-31T22:55:57.967 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66370.log.gz 2024-08-31T22:55:57.968 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.137787.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66521.log.gz 2024-08-31T22:55:57.968 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.137787.log.gz 2024-08-31T22:55:57.968 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61657.log.gz 2024-08-31T22:55:57.969 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61971.log.gz 2024-08-31T22:55:57.970 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62157.log.gz 2024-08-31T22:55:57.970 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62250.log.gz 2024-08-31T22:55:57.970 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67641.log.gz 2024-08-31T22:55:57.971 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55712.log.gz 2024-08-31T22:55:57.971 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.102316.log.gz 2024-08-31T22:55:57.972 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103378.log.gz 2024-08-31T22:55:57.973 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42515.log.gz 2024-08-31T22:55:57.974 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103529.log.gz 2024-08-31T22:55:57.974 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.115590.log.gz 2024-08-31T22:55:57.974 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.147495.log.gz 2024-08-31T22:55:57.974 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37781.log.gz 2024-08-31T22:55:57.975 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28037.log 2024-08-31T22:55:57.975 INFO:teuthology.orchestra.run.smithi032.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119038.log.gz 2024-08-31T22:55:57.976 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.93534.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57863.log 2024-08-31T22:55:57.976 INFO:teuthology.orchestra.run.smithi032.stderr: 19.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.29589.log.gz 2024-08-31T22:55:57.977 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.28037.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69172.log 2024-08-31T22:55:57.977 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.95778.log.gz 2024-08-31T22:55:57.978 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.57863.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.96057.log.gz 2024-08-31T22:55:57.978 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58890.log.gz 2024-08-31T22:55:57.978 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112868.log 2024-08-31T22:55:57.979 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.69172.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.155483.log 2024-08-31T22:55:57.980 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.112868.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110422.log 2024-08-31T22:55:57.982 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.155483.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117051.log.gz 2024-08-31T22:55:57.982 INFO:teuthology.orchestra.run.smithi032.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.155483.log.gz 2024-08-31T22:55:57.983 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160592.log 2024-08-31T22:55:57.983 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.110422.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.163698.log 2024-08-31T22:55:57.984 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.160592.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.160592.log.gz 2024-08-31T22:55:57.984 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109209.log 2024-08-31T22:55:57.985 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.163698.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.163698.log.gz 2024-08-31T22:55:57.985 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86156.log 2024-08-31T22:55:57.986 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.109209.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75661.log 2024-08-31T22:55:57.987 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.86156.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68110.log.gz 2024-08-31T22:55:57.987 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.162911.log 2024-08-31T22:55:57.988 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79432.log 2024-08-31T22:55:57.989 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77467.log.gz 2024-08-31T22:55:57.989 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.162911.log: 83.3% -- replaced with /var/log/ceph/cluster2-client.mirror.162911.log.gz 2024-08-31T22:55:57.989 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33332.log.gz 2024-08-31T22:55:57.989 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36750.log.gz 2024-08-31T22:55:57.991 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74835.log 2024-08-31T22:55:57.991 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.79432.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36692.log 2024-08-31T22:55:57.992 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.74835.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-08-31T22:55:57.993 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.151989.log.gz 2024-08-31T22:55:57.993 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.36692.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98522.log 2024-08-31T22:55:57.994 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59134.log 2024-08-31T22:55:57.996 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.98522.log: /var/log/ceph/cluster2-client.mirror.59134.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118231.log 2024-08-31T22:55:57.996 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155889.log 2024-08-31T22:55:57.996 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160495.log 2024-08-31T22:55:57.997 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.118231.log: /var/log/ceph/cluster1-client.mirror.155889.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65715.log 2024-08-31T22:55:57.998 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.160495.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.160495.log.gz 2024-08-31T22:55:58.000 INFO:teuthology.orchestra.run.smithi032.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.91388.log.gz 2024-08-31T22:55:58.000 INFO:teuthology.orchestra.run.smithi032.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.155889.log.gz 2024-08-31T22:55:58.000 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66428.log 2024-08-31T22:55:58.000 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152308.log 2024-08-31T22:55:58.001 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160996.log 2024-08-31T22:55:58.001 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.66428.log: /var/log/ceph/cluster1-client.mirror.152308.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.152308.log.gz 2024-08-31T22:55:58.001 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97177.log 2024-08-31T22:55:58.002 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73060.log 2024-08-31T22:55:58.002 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.160996.log: /var/log/ceph/cluster2-client.mirror.97177.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.160996.log.gz 2024-08-31T22:55:58.003 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93817.log 2024-08-31T22:55:58.003 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39366.log 2024-08-31T22:55:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.73060.log: /var/log/ceph/cluster2-client.mirror.93817.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.93534.log.gz 2024-08-31T22:55:58.005 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.107992.log.gz 2024-08-31T22:55:58.006 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.108368.log.gz 2024-08-31T22:55:58.006 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109244.log.gz 2024-08-31T22:55:58.006 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.109209.log.gz 2024-08-31T22:55:58.007 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.109488.log.gz 2024-08-31T22:55:58.007 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162526.log 2024-08-31T22:55:58.009 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.39366.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29232.log 2024-08-31T22:55:58.009 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162526.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59041.log 2024-08-31T22:55:58.009 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.162526.log.gz 2024-08-31T22:55:58.009 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.29232.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29232.log.gz 2024-08-31T22:55:58.009 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161558.log 2024-08-31T22:55:58.010 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.161581.log 2024-08-31T22:55:58.010 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.59041.log: /var/log/ceph/cluster1-client.mirror.161558.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.161558.log.gz 2024-08-31T22:55:58.010 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111616.log 2024-08-31T22:55:58.011 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96092.log 2024-08-31T22:55:58.011 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.161581.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.161581.log.gz 2024-08-31T22:55:58.011 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.111616.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99549.log 2024-08-31T22:55:58.012 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.96092.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31801.log 2024-08-31T22:55:58.012 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.99549.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117709.log 2024-08-31T22:55:58.012 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.31801.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.81637.log.gz 2024-08-31T22:55:58.017 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.134772.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158684.log 2024-08-31T22:55:58.017 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:58.017 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.117709.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162413.log 2024-08-31T22:55:58.017 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66614.log 2024-08-31T22:55:58.017 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158684.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102913.log 2024-08-31T22:55:58.017 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2%/var/log/ceph/cluster1-client.mirror.162413.log: -- replaced with /var/log/ceph/cluster2-client.mirror.158684.log.gz 2024-08-31T22:55:58.017 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.66614.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.162413.log.gz 2024-08-31T22:55:58.017 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102072.log 2024-08-31T22:55:58.017 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.102913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157799.log 2024-08-31T22:55:58.018 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32270.log 2024-08-31T22:55:58.018 INFO:teuthology.orchestra.run.smithi032.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.148259.log.gz 2024-08-31T22:55:58.018 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157799.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64033.log 2024-08-31T22:55:58.018 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.32270.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76126.log 2024-08-31T22:55:58.018 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.157799.log.gz 2024-08-31T22:55:58.019 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64033.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30495.log.gz 2024-08-31T22:55:58.019 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.102072.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68424.log 2024-08-31T22:55:58.019 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.76126.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65118.log 2024-08-31T22:55:58.019 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.68424.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143853.log 2024-08-31T22:55:58.020 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.65118.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110271.log 2024-08-31T22:55:58.020 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.143853.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51414.log.gz 2024-08-31T22:55:58.021 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63192.log 2024-08-31T22:55:58.021 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.110271.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157732.log 2024-08-31T22:55:58.022 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63192.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94189.log 2024-08-31T22:55:58.022 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157732.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119168.log 2024-08-31T22:55:58.022 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94189.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.55956.log.gz 2024-08-31T22:55:58.022 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.157732.log.gz 2024-08-31T22:55:58.023 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76498.log.gz 2024-08-31T22:55:58.023 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147361.log 2024-08-31T22:55:58.023 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119168.log: 88.4%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.86249.log.gz -5 2024-08-31T22:55:58.023 INFO:teuthology.orchestra.run.smithi032.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.163213.log 2024-08-31T22:55:58.024 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.147361.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97460.log 2024-08-31T22:55:58.024 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.147361.log.gz 2024-08-31T22:55:58.025 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.92883.log.gz 2024-08-31T22:55:58.025 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120115.log 2024-08-31T22:55:58.025 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.163213.log: /var/log/ceph/cluster2-client.mirror.97460.log: 83.6% -- replaced with /var/log/ceph/cluster2-client.mirror.163213.log.gz 2024-08-31T22:55:58.025 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89427.log 2024-08-31T22:55:58.025 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.120115.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98018.log 2024-08-31T22:55:58.026 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.89427.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128833.log 2024-08-31T22:55:58.026 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.98018.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63436.log 2024-08-31T22:55:58.026 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.128833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111337.log 2024-08-31T22:55:58.026 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63436.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85408.log.gz 2024-08-31T22:55:58.027 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101196.log 2024-08-31T22:55:58.027 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.111337.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135780.log 2024-08-31T22:55:58.027 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98336.log 2024-08-31T22:55:58.028 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.101196.log: /var/log/ceph/cluster2-client.mirror.135780.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135780.log.gz 2024-08-31T22:55:58.028 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.86962.log.gz 2024-08-31T22:55:58.028 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82947.log.gz 2024-08-31T22:55:58.028 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98522.log.gz 2024-08-31T22:55:58.028 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.98708.log.gz 2024-08-31T22:55:58.029 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63436.log.gz 2024-08-31T22:55:58.031 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117423.log 2024-08-31T22:55:58.031 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.98336.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97774.log 2024-08-31T22:55:58.033 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.87931.log.gz 2024-08-31T22:55:58.033 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.88210.log.gz 2024-08-31T22:55:58.033 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88396.log.gz 2024-08-31T22:55:58.035 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65773.log.gz 2024-08-31T22:55:58.035 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.117423.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65715.log.gz 2024-08-31T22:55:58.035 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64033.log.gz 2024-08-31T22:55:58.035 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65304.log.gz 2024-08-31T22:55:58.035 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74494.log.gz 2024-08-31T22:55:58.036 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.110422.log.gz 2024-08-31T22:55:58.036 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74835.log.gz 2024-08-31T22:55:58.036 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39366.log.gz 2024-08-31T22:55:58.036 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.94189.log.gz 2024-08-31T22:55:58.039 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140213.log 2024-08-31T22:55:58.039 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.97774.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120001.log 2024-08-31T22:55:58.040 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156257.log 2024-08-31T22:55:58.040 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.140213.log: /var/log/ceph/cluster2-client.mirror.120001.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59041.log.gz 2024-08-31T22:55:58.041 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116396.log.gz 2024-08-31T22:55:58.046 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51848.log.gz 2024-08-31T22:55:58.047 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144483.log 2024-08-31T22:55:58.048 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156257.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.156257.log.gz 2024-08-31T22:55:58.049 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67455.log 2024-08-31T22:55:58.049 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.144483.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.144483.log.gz 2024-08-31T22:55:58.050 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43356.log 2024-08-31T22:55:58.050 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67455.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52255.log 2024-08-31T22:55:58.051 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.43356.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157521.log 2024-08-31T22:55:58.051 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.52255.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50949.log 2024-08-31T22:55:58.053 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.157521.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.157521.log.gz 2024-08-31T22:55:58.053 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158300.log 2024-08-31T22:55:58.054 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.50949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118118.log 2024-08-31T22:55:58.054 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.158300.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.158300.log.gz 2024-08-31T22:55:58.055 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41674.log 2024-08-31T22:55:58.055 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.118118.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.58572.log.gz 2024-08-31T22:55:58.058 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107469.log 2024-08-31T22:55:58.059 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.41674.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165589.log 2024-08-31T22:55:58.059 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.107469.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82327.log.gz 2024-08-31T22:55:58.061 INFO:teuthology.orchestra.run.smithi032.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.165080.log.gz 2024-08-31T22:55:58.062 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79676.log 2024-08-31T22:55:58.062 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99270.log 2024-08-31T22:55:58.063 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165589.log: /var/log/ceph/cluster2-client.mirror.79676.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.165589.log.gz 2024-08-31T22:55:58.063 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46346.log.gz 2024-08-31T22:55:58.064 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85315.log 2024-08-31T22:55:58.064 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.155298.log 2024-08-31T22:55:58.064 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.99270.log: /var/log/ceph/cluster2-client.mirror.85315.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56332.log.gz 2024-08-31T22:55:58.065 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72405.log 2024-08-31T22:55:58.065 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48121.log 2024-08-31T22:55:58.066 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.155298.log: /var/log/ceph/cluster2-client.mirror.72405.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57863.log.gz 2024-08-31T22:55:58.066 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50318.log 2024-08-31T22:55:58.066 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.155298.log.gz 2024-08-31T22:55:58.067 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48121.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122475.log 2024-08-31T22:55:58.067 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.50318.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31801.log.gz 2024-08-31T22:55:58.067 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128702.log 2024-08-31T22:55:58.067 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.36692.log.gz 2024-08-31T22:55:58.068 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.122475.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67955.log.gz 2024-08-31T22:55:58.068 INFO:teuthology.orchestra.run.smithi032.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119365.log.gz 2024-08-31T22:55:58.075 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97460.log.gz 2024-08-31T22:55:58.075 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43763.log 2024-08-31T22:55:58.075 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.107469.log.gz 2024-08-31T22:55:58.075 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.110271.log.gz 2024-08-31T22:55:58.075 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94658.log 2024-08-31T22:55:58.076 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.128702.log: /var/log/ceph/cluster2-client.mirror.43763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148160.log 2024-08-31T22:55:58.077 INFO:teuthology.orchestra.run.smithi032.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.128702.log.gz 2024-08-31T22:55:58.077 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.94658.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38843.log 2024-08-31T22:55:58.077 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.148160.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.148160.log.gz 2024-08-31T22:55:58.077 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92263.log 2024-08-31T22:55:58.078 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.38843.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.118611.log.gz 2024-08-31T22:55:58.083 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30530.log.gz 2024-08-31T22:55:58.083 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34452.log.gz 2024-08-31T22:55:58.083 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166176.log 2024-08-31T22:55:58.084 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.92263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165630.log 2024-08-31T22:55:58.086 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.admin.166176.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166176.log.gz 2024-08-31T22:55:58.086 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34882.log.gz 2024-08-31T22:55:58.086 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.35293.log.gz 2024-08-31T22:55:58.086 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.92263.log.gz 2024-08-31T22:55:58.090 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.116803.log.gz 2024-08-31T22:55:58.103 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.83412.log.gz 2024-08-31T22:55:58.104 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86811.log.gz 2024-08-31T22:55:58.104 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74180.log 2024-08-31T22:55:58.104 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.165630.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90515.log 2024-08-31T22:55:58.105 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.74180.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94972.log.gz 2024-08-31T22:55:58.105 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31464.log 2024-08-31T22:55:58.105 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.90515.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.165630.log.gz 2024-08-31T22:55:58.106 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64967.log 2024-08-31T22:55:58.106 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.31464.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67862.log 2024-08-31T22:55:58.111 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64967.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141557.log 2024-08-31T22:55:58.111 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.67862.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93441.log 2024-08-31T22:55:58.127 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.141557.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62064.log 2024-08-31T22:55:58.139 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.93441.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43298.log 2024-08-31T22:55:58.144 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.62064.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129275.log.gz 2024-08-31T22:55:58.144 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48214.log 2024-08-31T22:55:58.145 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.43298.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100355.log 2024-08-31T22:55:58.145 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.48214.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58014.log 2024-08-31T22:55:58.146 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.100355.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36847.log 2024-08-31T22:55:58.147 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58014.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83168.log 2024-08-31T22:55:58.147 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.36847.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41353.log 2024-08-31T22:55:58.148 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.83168.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88551.log 2024-08-31T22:55:58.148 INFO:teuthology.orchestra.run.smithi032.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.147109.log.gz 2024-08-31T22:55:58.149 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.41353.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127815.log 2024-08-31T22:55:58.149 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.88551.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29659.log 2024-08-31T22:55:58.149 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.127815.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.164825.log 2024-08-31T22:55:58.150 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.29659.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90495.log 2024-08-31T22:55:58.151 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.164825.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.164825.log.gz 2024-08-31T22:55:58.151 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75754.log 2024-08-31T22:55:58.152 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.90495.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87055.log 2024-08-31T22:55:58.152 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.75754.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147428.log 2024-08-31T22:55:58.153 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.87055.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58328.log 2024-08-31T22:55:58.154 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.147428.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.147428.log.gz 2024-08-31T22:55:58.157 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162481.log 2024-08-31T22:55:58.157 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58328.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139536.log 2024-08-31T22:55:58.157 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162481.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.162481.log.gz 2024-08-31T22:55:58.157 INFO:teuthology.orchestra.run.smithi032.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.140213.log.gz 2024-08-31T22:55:58.157 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.165449.log 2024-08-31T22:55:58.157 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.139536.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.139536.log.gz 2024-08-31T22:55:58.158 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64560.log 2024-08-31T22:55:58.158 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.165449.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54123.log.gz 2024-08-31T22:55:58.158 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.165449.log.gz 2024-08-31T22:55:58.159 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66273.log.gz 2024-08-31T22:55:58.159 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72405.log.gz 2024-08-31T22:55:58.159 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.53096.log.gz 2024-08-31T22:55:58.159 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72347.log.gz 2024-08-31T22:55:58.159 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74180.log.gz 2024-08-31T22:55:58.160 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.165930.log 2024-08-31T22:55:58.160 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose 88.6% -- /var/log/ceph/cluster2-client.mirror.59506.log -- replaced with /var/log/ceph/cluster2-client.mirror.74556.log.gz 2024-08-31T22:55:58.160 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:55:58.161 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.64560.log: /var/log/ceph/cluster2-client.mirror.165930.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.165930.log.gz 2024-08-31T22:55:58.162 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37254.log 2024-08-31T22:55:58.163 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79304.log 2024-08-31T22:55:58.163 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.59506.log: /var/log/ceph/cluster2-client.mirror.37254.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51135.log 2024-08-31T22:55:58.164 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114608.log 2024-08-31T22:55:58.164 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.79304.log: /var/log/ceph/cluster2-client.mirror.51135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47687.log 2024-08-31T22:55:58.164 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55022.log 2024-08-31T22:55:58.165 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.114608.log: /var/log/ceph/cluster2-client.mirror.47687.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114794.log 2024-08-31T22:55:58.165 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106535.log 2024-08-31T22:55:58.165 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.55022.log: 58.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29181.log.gz 2024-08-31T22:55:58.166 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.114794.log: 80.0% -- replaced with /var/log/ceph/cluster2-client.admin.29284.log.gz 2024-08-31T22:55:58.166 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120803.log 2024-08-31T22:55:58.166 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.106535.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34696.log 2024-08-31T22:55:58.167 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104291.log 2024-08-31T22:55:58.167 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.34696.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102444.log.gz 2024-08-31T22:55:58.167 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.120803.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44104.log 2024-08-31T22:55:58.168 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53716.log 2024-08-31T22:55:58.168 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.104291.log: /var/log/ceph/cluster2-client.mirror.44104.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117365.log 2024-08-31T22:55:58.169 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156976.log 2024-08-31T22:55:58.169 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53716.log: /var/log/ceph/cluster2-client.mirror.117365.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.101138.log.gz 2024-08-31T22:55:58.170 INFO:teuthology.orchestra.run.smithi032.stderr: 3.5% -- replaced with /var/log/ceph/cluster2-client.mirror.29352.log.gz 2024-08-31T22:55:58.171 INFO:teuthology.orchestra.run.smithi032.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.120001.log.gz 2024-08-31T22:55:58.171 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42794.log 2024-08-31T22:55:58.172 INFO:teuthology.orchestra.run.smithi032.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.29659.log.gz 2024-08-31T22:55:58.172 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30437.log 2024-08-31T22:55:58.172 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.156976.log: /var/log/ceph/cluster2-client.mirror.42794.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119907.log.gz 2024-08-31T22:55:58.173 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31464.log.gz 2024-08-31T22:55:58.173 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.94658.log.gz 2024-08-31T22:55:58.174 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.95813.log.gz 2024-08-31T22:55:58.174 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59506.log.gz 2024-08-31T22:55:58.174 INFO:teuthology.orchestra.run.smithi032.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.156976.log.gz 2024-08-31T22:55:58.175 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91759.log 2024-08-31T22:55:58.175 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124846.log 2024-08-31T22:55:58.175 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.30437.log: /var/log/ceph/cluster2-client.mirror.91759.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.79246.log.gz 2024-08-31T22:55:58.176 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.79432.log.gz 2024-08-31T22:55:58.176 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58421.log.gz 2024-08-31T22:55:58.176 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55398.log.gz 2024-08-31T22:55:58.176 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70350.log.gz 2024-08-31T22:55:58.177 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32828.log.gz 2024-08-31T22:55:58.178 INFO:teuthology.orchestra.run.smithi032.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.118075.log.gz 2024-08-31T22:55:58.178 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98301.log 2024-08-31T22:55:58.179 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45131.log 2024-08-31T22:55:58.179 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.124846.log: /var/log/ceph/cluster2-client.mirror.98301.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124846.log.gz 2024-08-31T22:55:58.179 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119168.log.gz 2024-08-31T22:55:58.180 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41767.log 2024-08-31T22:55:58.180 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91446.log 2024-08-31T22:55:58.180 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.45131.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119449.log.gz 2024-08-31T22:55:58.181 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29147.log 2024-08-31T22:55:58.181 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.41767.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.93348.log.gz 2024-08-31T22:55:58.181 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.91446.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.107748.log.gz 2024-08-31T22:55:58.181 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.109302.log.gz 2024-08-31T22:55:58.182 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81858.log 2024-08-31T22:55:58.182 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97119.log 2024-08-31T22:55:58.183 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.29147.log: /var/log/ceph/cluster2-client.mirror.81858.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.112364.log.gz 2024-08-31T22:55:58.183 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156208.log 2024-08-31T22:55:58.183 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160049.log 2024-08-31T22:55:58.184 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.97119.log: 80.3%/var/log/ceph/cluster2-client.mirror.156208.log: -- replaced with /var/log/ceph/cluster2-client.admin.28037.log.gz 2024-08-31T22:55:58.184 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.34696.log.gz 2024-08-31T22:55:58.184 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.156208.log.gz 2024-08-31T22:55:58.185 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30437.log.gz 2024-08-31T22:55:58.186 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41895.log 2024-08-31T22:55:58.186 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.160049.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162841.log 2024-08-31T22:55:58.187 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.41895.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.160049.log.gz 2024-08-31T22:55:58.187 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78556.log 2024-08-31T22:55:58.188 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162841.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113860.log 2024-08-31T22:55:58.188 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.162841.log.gz 2024-08-31T22:55:58.189 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.78556.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36564.log 2024-08-31T22:55:58.189 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.76967.log.gz 2024-08-31T22:55:58.189 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.113860.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57487.log 2024-08-31T22:55:58.189 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.36564.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52445.log 2024-08-31T22:55:58.190 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.57487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148192.log 2024-08-31T22:55:58.190 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.52445.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53565.log 2024-08-31T22:55:58.190 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.148192.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162143.log 2024-08-31T22:55:58.190 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53565.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148192.log.gz 2024-08-31T22:55:58.191 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-08-31T22:55:58.191 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162143.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40525.log 2024-08-31T22:55:58.191 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.162143.log.gz/var/log/ceph/cluster2-osd.2.log: 2024-08-31T22:55:58.192 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91203.log 2024-08-31T22:55:58.192 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.40525.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.165568.log 2024-08-31T22:55:58.193 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.91203.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56181.log 2024-08-31T22:55:58.193 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.165568.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.29517.log 2024-08-31T22:55:58.193 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.56181.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.165568.log.gz 2024-08-31T22:55:58.194 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55584.log 2024-08-31T22:55:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.1.29517.log: /var/log/ceph/cluster2-client.mirror.55584.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99700.log 2024-08-31T22:55:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32456.log 2024-08-31T22:55:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.99700.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77432.log 2024-08-31T22:55:58.196 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.90863.log.gz 2024-08-31T22:55:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77432.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109864.log 2024-08-31T22:55:58.197 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51476.log 2024-08-31T22:55:58.206 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.32456.log: /var/log/ceph/cluster2-client.mirror.109864.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114701.log 2024-08-31T22:55:58.207 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.51476.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150031.log 2024-08-31T22:55:58.207 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42794.log.gz 2024-08-31T22:55:58.207 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.114701.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60068.log 2024-08-31T22:55:58.208 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.150031.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30125.log 2024-08-31T22:55:58.209 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.150031.log.gz 2024-08-31T22:55:58.209 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.60068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155032.log 2024-08-31T22:55:58.209 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.30125.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148595.log 2024-08-31T22:55:58.209 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147067.log 2024-08-31T22:55:58.211 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.155032.log: /var/log/ceph/cluster2-client.mirror.148595.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.155032.log.gz 2024-08-31T22:55:58.212 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.98301.log.gz 2024-08-31T22:55:58.212 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.91759.log.gz 2024-08-31T22:55:58.212 INFO:teuthology.orchestra.run.smithi032.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.148595.log.gz 2024-08-31T22:55:58.212 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160395.log 2024-08-31T22:55:58.213 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.147067.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.admin.91446.log.gz 2024-08-31T22:55:58.214 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100076.log 2024-08-31T22:55:58.214 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.160395.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.147067.log.gz 2024-08-31T22:55:58.215 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53623.log 2024-08-31T22:55:58.216 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.100076.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.160395.log.gz 2024-08-31T22:55:58.216 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54499.log 2024-08-31T22:55:58.217 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134707.log 2024-08-31T22:55:58.218 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53623.log: /var/log/ceph/cluster2-client.mirror.54499.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53286.log 2024-08-31T22:55:58.218 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.134707.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100169.log 2024-08-31T22:55:58.218 INFO:teuthology.orchestra.run.smithi032.stderr: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134707.log.gz 2024-08-31T22:55:58.219 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112457.log 2024-08-31T22:55:58.219 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.53286.log: /var/log/ceph/cluster2-client.mirror.100169.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122031.log 2024-08-31T22:55:58.219 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77529.log 2024-08-31T22:55:58.220 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.112457.log: /var/log/ceph/cluster2-client.mirror.122031.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61506.log 2024-08-31T22:55:58.220 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.77529.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45166.log 2024-08-31T22:55:58.221 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.61506.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103846.log 2024-08-31T22:55:58.222 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.45166.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66742.log 2024-08-31T22:55:58.223 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.admin.103846.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.162436.log 2024-08-31T22:55:58.224 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.66742.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92825.log 2024-08-31T22:55:58.225 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.162436.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.162436.log.gz 2024-08-31T22:55:58.227 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100855.log 2024-08-31T22:55:58.228 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.92825.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111127.log 2024-08-31T22:55:58.230 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.100855.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33762.log 2024-08-31T22:55:58.232 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45131.log.gz 2024-08-31T22:55:58.232 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45166.log.gz 2024-08-31T22:55:58.232 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster1-client.mirror.111127.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.79304.log.gz 2024-08-31T22:55:58.233 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53623.log.gz 2024-08-31T22:55:58.234 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.54499.log.gz 2024-08-31T22:55:58.234 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.96747.log.gz 2024-08-31T22:55:58.235 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73525.log 2024-08-31T22:55:58.235 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97739.log.gz 2024-08-31T22:55:58.235 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.97774.log.gz 2024-08-31T22:55:58.236 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.33762.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113147.log 2024-08-31T22:55:58.236 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.98336.log.gz 2024-08-31T22:55:58.237 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.53716.log.gz 2024-08-31T22:55:58.237 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.73525.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58855.log 2024-08-31T22:55:58.238 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.113147.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60909.log.gz 2024-08-31T22:55:58.238 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71227.log 2024-08-31T22:55:58.238 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64560.log.gz 2024-08-31T22:55:58.239 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74649.log.gz 2024-08-31T22:55:58.239 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100855.log.gz 2024-08-31T22:55:58.239 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96619.log.gz 2024-08-31T22:55:58.239 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.58855.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100076.log.gz 2024-08-31T22:55:58.240 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39649.log.gz 2024-08-31T22:55:58.240 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55247.log 2024-08-31T22:55:58.241 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.71227.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47842.log 2024-08-31T22:55:58.241 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.55247.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105225.log 2024-08-31T22:55:58.242 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.47842.log: 19.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.29517.log.gz 2024-08-31T22:55:58.242 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74591.log 2024-08-31T22:55:58.242 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.105225.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34045.log 2024-08-31T22:55:58.243 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.74591.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.113147.log.gz 2024-08-31T22:55:58.243 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63568.log 2024-08-31T22:55:58.244 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.80738.log.gz 2024-08-31T22:55:58.245 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.34045.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116245.log 2024-08-31T22:55:58.245 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.63568.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83261.log.gz 2024-08-31T22:55:58.246 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43763.log.gz 2024-08-31T22:55:58.246 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119256.log 2024-08-31T22:55:58.246 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.116245.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44139.log.gz 2024-08-31T22:55:58.247 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47780.log.gz 2024-08-31T22:55:58.247 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47842.log.gz 2024-08-31T22:55:58.247 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49462.log.gz 2024-08-31T22:55:58.248 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.77432.log.gz 2024-08-31T22:55:58.248 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.77529.log.gz 2024-08-31T22:55:58.248 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cluster2-client.mirror.119256.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41767.log.gz 2024-08-31T22:55:58.249 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41895.log.gz 2024-08-31T22:55:58.251 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32022.log.gz 2024-08-31T22:55:58.252 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36564.log.gz 2024-08-31T22:55:58.253 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.81858.log.gz 2024-08-31T22:55:58.253 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.78556.log.gz 2024-08-31T22:55:58.257 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34045.log.gz 2024-08-31T22:55:58.257 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.93069.log.gz 2024-08-31T22:55:58.257 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.81916.log.gz 2024-08-31T22:55:58.257 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42643.log.gz 2024-08-31T22:55:58.257 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85842.log.gz 2024-08-31T22:55:58.257 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33890.log.gz 2024-08-31T22:55:58.257 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.57921.log.gz 2024-08-31T22:55:58.262 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.113860.log.gz 2024-08-31T22:55:58.262 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.117237.log.gz 2024-08-31T22:55:58.265 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.116861.log.gz 2024-08-31T22:55:58.271 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.143853.log.gz 2024-08-31T22:55:58.290 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57115.log.gz 2024-08-31T22:55:58.291 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63568.log.gz 2024-08-31T22:55:58.291 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45259.log.gz 2024-08-31T22:55:58.291 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60851.log.gz 2024-08-31T22:55:58.291 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73525.log.gz 2024-08-31T22:55:58.291 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.79618.log.gz 2024-08-31T22:55:58.293 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74029.log.gz 2024-08-31T22:55:58.294 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38374.log.gz 2024-08-31T22:55:58.294 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39273.log.gz 2024-08-31T22:55:58.294 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105225.log.gz 2024-08-31T22:55:58.295 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74591.log.gz 2024-08-31T22:55:58.295 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30960.log.gz 2024-08-31T22:55:58.295 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104163.log.gz 2024-08-31T22:55:58.295 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119256.log.gz 2024-08-31T22:55:58.295 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50949.log.gz 2024-08-31T22:55:58.299 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58855.log.gz 2024-08-31T22:55:58.300 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84567.log.gz 2024-08-31T22:55:58.300 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.89427.log.gz 2024-08-31T22:55:58.301 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.89764.log.gz 2024-08-31T22:55:58.301 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.116245.log.gz 2024-08-31T22:55:58.304 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50024.log.gz 2024-08-31T22:55:58.311 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44046.log.gz 2024-08-31T22:55:58.311 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45916.log.gz 2024-08-31T22:55:58.324 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.119389.log.gz 2024-08-31T22:55:58.324 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.119345.log.gz 2024-08-31T22:55:58.325 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.141557.log.gz 2024-08-31T22:55:58.328 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.49148.log.gz 2024-08-31T22:55:58.329 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.51976.log.gz 2024-08-31T22:55:58.330 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99270.log.gz 2024-08-31T22:55:58.330 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56425.log.gz 2024-08-31T22:55:58.332 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67862.log.gz 2024-08-31T22:55:58.332 INFO:teuthology.orchestra.run.smithi032.stderr: 55.3% -- replaced with /var/log/ceph/cluster1-client.mirror.29113.log.gz 2024-08-31T22:55:58.333 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.88923.log.gz 2024-08-31T22:55:58.333 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.55433.log.gz 2024-08-31T22:55:58.333 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77153.log.gz 2024-08-31T22:55:58.334 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36847.log.gz 2024-08-31T22:55:58.335 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55770.log.gz 2024-08-31T22:55:58.336 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31278.log.gz 2024-08-31T22:55:58.339 INFO:teuthology.orchestra.run.smithi032.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.128343.log.gz 2024-08-31T22:55:58.339 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75471.log.gz 2024-08-31T22:55:58.346 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46253.log.gz 2024-08-31T22:55:58.348 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46943.log.gz 2024-08-31T22:55:58.348 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.108647.log.gz 2024-08-31T22:55:58.350 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111616.log.gz 2024-08-31T22:55:58.350 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.78742.log.gz 2024-08-31T22:55:58.351 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37254.log.gz 2024-08-31T22:55:58.351 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33704.log.gz 2024-08-31T22:55:58.352 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75661.log.gz 2024-08-31T22:55:58.364 INFO:teuthology.orchestra.run.smithi032.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.122031.log.gz 2024-08-31T22:55:58.364 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42422.log.gz 2024-08-31T22:55:58.367 INFO:teuthology.orchestra.run.smithi032.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118548.log.gz 2024-08-31T22:55:58.369 INFO:teuthology.orchestra.run.smithi032.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.118568.log.gz 2024-08-31T22:55:58.376 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87989.log.gz 2024-08-31T22:55:58.377 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.54313.log.gz 2024-08-31T22:55:58.377 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56739.log.gz 2024-08-31T22:55:58.377 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60758.log.gz 2024-08-31T22:55:58.377 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64370.log.gz 2024-08-31T22:55:58.380 INFO:teuthology.orchestra.run.smithi032.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128833.log.gz 2024-08-31T22:55:58.385 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71564.log.gz 2024-08-31T22:55:58.385 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73060.log.gz 2024-08-31T22:55:58.386 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74087.log.gz 2024-08-31T22:55:58.392 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104256.log.gz 2024-08-31T22:55:58.392 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.107000.log.gz 2024-08-31T22:55:58.400 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.115555.log.gz 2024-08-31T22:55:58.400 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38246.log.gz 2024-08-31T22:55:58.400 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39684.log.gz 2024-08-31T22:55:58.400 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31146.log.gz 2024-08-31T22:55:58.400 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31522.log.gz 2024-08-31T22:55:58.402 INFO:teuthology.orchestra.run.smithi032.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122475.log.gz 2024-08-31T22:55:58.408 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.116152.log.gz 2024-08-31T22:55:58.423 INFO:teuthology.orchestra.run.smithi032.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62772.log.gz 2024-08-31T22:55:58.433 INFO:teuthology.orchestra.run.smithi032.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.91319.log.gz 2024-08-31T22:55:58.436 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64874.log.gz 2024-08-31T22:55:58.438 INFO:teuthology.orchestra.run.smithi032.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.120623.log.gz 2024-08-31T22:55:58.444 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.101289.log.gz 2024-08-31T22:55:58.446 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.75371.log.gz 2024-08-31T22:55:58.446 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.78591.log.gz 2024-08-31T22:55:58.447 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.86307.log.gz 2024-08-31T22:55:58.448 INFO:teuthology.orchestra.run.smithi032.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.75214.log.gz 2024-08-31T22:55:58.448 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.98018.log.gz 2024-08-31T22:55:58.462 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63192.log.gz 2024-08-31T22:55:58.463 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63285.log.gz 2024-08-31T22:55:58.463 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66428.log.gz 2024-08-31T22:55:58.463 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66614.log.gz 2024-08-31T22:55:58.463 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69172.log.gz 2024-08-31T22:55:58.463 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.53565.log.gz 2024-08-31T22:55:58.473 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101196.log.gz 2024-08-31T22:55:58.474 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102072.log.gz 2024-08-31T22:55:58.476 INFO:teuthology.orchestra.run.smithi032.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119837.log.gz 2024-08-31T22:55:58.478 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.111337.log.gz 2024-08-31T22:55:58.478 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.116431.log.gz 2024-08-31T22:55:58.478 INFO:teuthology.orchestra.run.smithi032.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111127.log.gz 2024-08-31T22:55:58.479 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58514.log.gz 2024-08-31T22:55:58.479 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68331.log.gz 2024-08-31T22:55:58.479 INFO:teuthology.orchestra.run.smithi032.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.90558.log.gz 2024-08-31T22:55:58.482 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70013.log.gz 2024-08-31T22:55:58.482 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77374.log.gz 2024-08-31T22:55:58.486 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32270.log.gz 2024-08-31T22:55:58.487 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32735.log.gz 2024-08-31T22:55:58.487 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33204.log.gz 2024-08-31T22:55:58.487 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.127815.log.gz 2024-08-31T22:55:58.496 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.93817.log.gz 2024-08-31T22:55:58.496 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.107093.log.gz 2024-08-31T22:55:58.497 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.107562.log.gz 2024-08-31T22:55:58.504 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.112457.log.gz 2024-08-31T22:55:58.506 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30623.log.gz 2024-08-31T22:55:58.508 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75696.log.gz 2024-08-31T22:55:58.508 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75754.log.gz 2024-08-31T22:55:58.509 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.77002.log.gz 2024-08-31T22:55:58.511 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86156.log.gz 2024-08-31T22:55:58.512 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114360.log.gz 2024-08-31T22:55:58.513 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.87055.log.gz 2024-08-31T22:55:58.521 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.79583.log.gz 2024-08-31T22:55:58.522 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65211.log.gz 2024-08-31T22:55:58.522 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61471.log.gz 2024-08-31T22:55:58.534 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96526.log.gz 2024-08-31T22:55:58.535 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100355.log.gz 2024-08-31T22:55:58.536 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50821.log.gz 2024-08-31T22:55:58.536 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.84381.log.gz 2024-08-31T22:55:58.543 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.48435.log.gz 2024-08-31T22:55:58.543 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48586.log.gz 2024-08-31T22:55:58.543 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58014.log.gz 2024-08-31T22:55:58.544 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58328.log.gz 2024-08-31T22:55:58.567 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41674.log.gz 2024-08-31T22:55:58.567 INFO:teuthology.orchestra.run.smithi032.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75190.log.gz 2024-08-31T22:55:58.567 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.62844.log.gz 2024-08-31T22:55:58.569 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64746.log.gz 2024-08-31T22:55:58.569 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64967.log.gz 2024-08-31T22:55:58.572 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67114.log.gz 2024-08-31T22:55:58.572 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.55022.log.gz 2024-08-31T22:55:58.573 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99549.log.gz 2024-08-31T22:55:58.573 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32080.log.gz 2024-08-31T22:55:58.573 INFO:teuthology.orchestra.run.smithi032.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.118838.log.gz 2024-08-31T22:55:58.581 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.93441.log.gz 2024-08-31T22:55:58.596 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.114608.log.gz 2024-08-31T22:55:58.596 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114794.log.gz 2024-08-31T22:55:58.597 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116768.log.gz 2024-08-31T22:55:58.598 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33762.log.gz 2024-08-31T22:55:58.598 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76126.log.gz 2024-08-31T22:55:58.604 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.83168.log.gz 2024-08-31T22:55:58.609 INFO:teuthology.orchestra.run.smithi032.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118591.log.gz 2024-08-31T22:55:58.634 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57487.log.gz 2024-08-31T22:55:58.635 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63847.log.gz 2024-08-31T22:55:58.644 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.88551.log.gz 2024-08-31T22:55:58.645 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51135.log.gz 2024-08-31T22:55:58.645 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53286.log.gz 2024-08-31T22:55:58.645 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64467.log.gz 2024-08-31T22:55:58.645 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67455.log.gz 2024-08-31T22:55:58.646 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74122.log.gz 2024-08-31T22:55:58.646 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104105.log.gz 2024-08-31T22:55:58.646 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.104291.log.gz 2024-08-31T22:55:58.646 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.96371.log.gz 2024-08-31T22:55:58.646 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.99700.log.gz 2024-08-31T22:55:58.650 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.79056.log.gz 2024-08-31T22:55:58.650 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.116582.log.gz 2024-08-31T22:55:58.651 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43356.log.gz 2024-08-31T22:55:58.651 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43542.log.gz 2024-08-31T22:55:58.651 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48121.log.gz 2024-08-31T22:55:58.651 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68424.log.gz 2024-08-31T22:55:58.683 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110608.log.gz 2024-08-31T22:55:58.689 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.80273.log.gz 2024-08-31T22:55:58.699 INFO:teuthology.orchestra.run.smithi032.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.91362.log.gz 2024-08-31T22:55:58.699 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32456.log.gz 2024-08-31T22:55:58.718 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.85036.log.gz 2024-08-31T22:55:58.718 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.85280.log.gz 2024-08-31T22:55:58.718 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.85315.log.gz 2024-08-31T22:55:58.718 INFO:teuthology.orchestra.run.smithi032.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118861.log.gz 2024-08-31T22:55:58.728 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.107527.log.gz 2024-08-31T22:55:58.734 INFO:teuthology.orchestra.run.smithi032.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.118478.log.gz 2024-08-31T22:55:58.735 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114701.log.gz 2024-08-31T22:55:58.745 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75975.log.gz 2024-08-31T22:55:58.748 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.97177.log.gz 2024-08-31T22:55:58.751 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52255.log.gz 2024-08-31T22:55:58.754 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.34545.log.gz 2024-08-31T22:55:58.778 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63661.log.gz 2024-08-31T22:55:58.784 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64839.log.gz 2024-08-31T22:55:58.784 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71227.log.gz 2024-08-31T22:55:58.786 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74273.log.gz 2024-08-31T22:55:58.787 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.102037.log.gz 2024-08-31T22:55:58.788 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106349.log.gz 2024-08-31T22:55:58.788 INFO:teuthology.orchestra.run.smithi032.stderr: 3.5% -- replaced with /var/log/ceph/cluster1-client.mirror.29147.log.gz 2024-08-31T22:55:58.788 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.94224.log.gz 2024-08-31T22:55:58.789 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.112868.log.gz 2024-08-31T22:55:58.796 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37126.log.gz 2024-08-31T22:55:58.797 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40525.log.gz 2024-08-31T22:55:58.798 INFO:teuthology.orchestra.run.smithi032.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75237.log.gz 2024-08-31T22:55:58.798 INFO:teuthology.orchestra.run.smithi032.stderr: 97.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41279.log.gz 2024-08-31T22:55:58.803 INFO:teuthology.orchestra.run.smithi032.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.118388.log.gz 2024-08-31T22:55:58.806 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51476.log.gz 2024-08-31T22:55:58.824 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55584.log.gz 2024-08-31T22:55:58.825 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36599.log.gz 2024-08-31T22:55:58.852 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.117423.log.gz 2024-08-31T22:55:58.852 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117458.log.gz 2024-08-31T22:55:58.852 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.117551.log.gz 2024-08-31T22:55:58.853 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.117982.log.gz 2024-08-31T22:55:58.853 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97119.log.gz 2024-08-31T22:55:58.854 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.97681.log.gz 2024-08-31T22:55:58.878 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65118.log.gz 2024-08-31T22:55:58.879 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.79676.log.gz 2024-08-31T22:55:58.879 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53154.log.gz 2024-08-31T22:55:58.879 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.61506.log.gz 2024-08-31T22:55:58.879 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106872.log.gz 2024-08-31T22:55:58.879 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100448.log.gz 2024-08-31T22:55:58.879 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59134.log.gz 2024-08-31T22:55:58.886 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.116020.log.gz 2024-08-31T22:55:58.902 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43949.log.gz 2024-08-31T22:55:58.902 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47687.log.gz 2024-08-31T22:55:58.904 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48214.log.gz 2024-08-31T22:55:58.904 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48811.log.gz 2024-08-31T22:55:58.918 INFO:teuthology.orchestra.run.smithi032.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62726.log.gz 2024-08-31T22:55:58.933 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55247.log.gz 2024-08-31T22:55:58.934 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33111.log.gz 2024-08-31T22:55:58.934 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40897.log.gz 2024-08-31T22:55:58.934 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36192.log.gz 2024-08-31T22:55:58.944 INFO:teuthology.orchestra.run.smithi032.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.118231.log.gz 2024-08-31T22:55:58.947 INFO:teuthology.orchestra.run.smithi032.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.90584.log.gz 2024-08-31T22:55:58.951 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.78463.log.gz 2024-08-31T22:55:58.951 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34266.log.gz 2024-08-31T22:55:58.968 INFO:teuthology.orchestra.run.smithi007.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-08-31T22:55:58.986 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.117365.log.gz 2024-08-31T22:55:59.003 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60068.log.gz 2024-08-31T22:55:59.005 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.admin.103846.log.gz 2024-08-31T22:55:59.006 INFO:teuthology.orchestra.run.smithi032.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.29577.log.gz 2024-08-31T22:55:59.032 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102913.log.gz 2024-08-31T22:55:59.032 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106907.log.gz 2024-08-31T22:55:59.032 INFO:teuthology.orchestra.run.smithi032.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66742.log.gz 2024-08-31T22:55:59.040 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.92825.log.gz 2024-08-31T22:55:59.040 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96092.log.gz 2024-08-31T22:55:59.044 INFO:teuthology.orchestra.run.smithi032.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41233.log.gz 2024-08-31T22:55:59.060 INFO:teuthology.orchestra.run.smithi032.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41257.log.gz 2024-08-31T22:55:59.118 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52445.log.gz 2024-08-31T22:55:59.143 INFO:teuthology.orchestra.run.smithi032.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118188.log.gz 2024-08-31T22:55:59.180 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62064.log.gz 2024-08-31T22:55:59.181 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106535.log.gz 2024-08-31T22:55:59.182 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100169.log.gz 2024-08-31T22:55:59.190 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44104.log.gz 2024-08-31T22:55:59.196 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48997.log.gz 2024-08-31T22:55:59.226 INFO:teuthology.orchestra.run.smithi032.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.103722.log.gz 2024-08-31T22:55:59.250 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38843.log.gz 2024-08-31T22:55:59.251 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.41353.log.gz 2024-08-31T22:55:59.279 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.112120.log.gz 2024-08-31T22:55:59.322 INFO:teuthology.orchestra.run.smithi032.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.118118.log.gz 2024-08-31T22:55:59.323 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47094.log.gz 2024-08-31T22:55:59.343 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.94600.log.gz 2024-08-31T22:55:59.407 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81079.log.gz 2024-08-31T22:55:59.408 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81358.log.gz 2024-08-31T22:55:59.409 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.109709.log.gz 2024-08-31T22:55:59.409 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.109864.log.gz 2024-08-31T22:55:59.490 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52069.log.gz 2024-08-31T22:55:59.491 INFO:teuthology.orchestra.run.smithi032.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115213.log.gz 2024-08-31T22:55:59.573 INFO:teuthology.orchestra.run.smithi032.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.91203.log.gz 2024-08-31T22:55:59.590 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38467.log.gz 2024-08-31T22:55:59.591 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39215.log.gz 2024-08-31T22:55:59.593 INFO:teuthology.orchestra.run.smithi032.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44290.log.gz 2024-08-31T22:55:59.634 INFO:teuthology.orchestra.run.smithi032.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61843.log.gz 2024-08-31T22:55:59.634 INFO:teuthology.orchestra.run.smithi032.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62683.log.gz 2024-08-31T22:55:59.655 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43298.log.gz 2024-08-31T22:55:59.681 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56181.log.gz 2024-08-31T22:55:59.682 INFO:teuthology.orchestra.run.smithi032.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56274.log.gz 2024-08-31T22:55:59.706 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.108124.log.gz 2024-08-31T22:55:59.796 INFO:teuthology.orchestra.run.smithi032.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.117752.log.gz 2024-08-31T22:56:00.155 INFO:teuthology.orchestra.run.smithi032.stderr: 93.5% 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.117709.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.115233.log.gz 2024-08-31T22:56:00.156 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:56:00.635 INFO:teuthology.orchestra.run.smithi032.stderr: 93.2% 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90515.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.75257.log.gz 2024-08-31T22:56:00.635 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T22:56:00.635 INFO:teuthology.orchestra.run.smithi032.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.117685.log.gz 93.4% 2024-08-31T22:56:00.636 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.91299.log.gz 2024-08-31T22:56:00.636 INFO:teuthology.orchestra.run.smithi032.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50251.log.gz 2024-08-31T22:56:00.636 INFO:teuthology.orchestra.run.smithi032.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.90495.log.gz 2024-08-31T22:56:01.151 INFO:teuthology.orchestra.run.smithi032.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99456.log.gz 2024-08-31T22:56:01.154 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.115869.log.gz 2024-08-31T22:56:01.289 INFO:teuthology.orchestra.run.smithi032.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50318.log.gz 2024-08-31T22:56:02.701 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115390.log.gz 2024-08-31T22:56:03.320 INFO:teuthology.orchestra.run.smithi032.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.1.29645.log.gz 2024-08-31T22:56:04.176 INFO:teuthology.orchestra.run.smithi032.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.120962.log.gz 2024-08-31T22:56:05.890 INFO:teuthology.orchestra.run.smithi032.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119860.log.gz 2024-08-31T22:56:07.454 INFO:teuthology.orchestra.run.smithi032.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.119791.log.gz 2024-08-31T22:56:07.658 INFO:teuthology.orchestra.run.smithi032.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.29577.log.gz 2024-08-31T22:56:08.578 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62903.log.gz 2024-08-31T22:56:09.077 INFO:teuthology.orchestra.run.smithi032.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119565.log.gz 2024-08-31T22:56:09.250 INFO:teuthology.orchestra.run.smithi032.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.115282.log.gz 2024-08-31T22:56:09.341 INFO:teuthology.orchestra.run.smithi032.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.120548.log.gz 2024-08-31T22:56:10.271 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.111269.log.gz 2024-08-31T22:56:11.407 INFO:teuthology.orchestra.run.smithi032.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111172.log.gz 2024-08-31T22:56:11.595 INFO:teuthology.orchestra.run.smithi032.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.120803.log.gz 2024-08-31T22:56:12.050 INFO:teuthology.orchestra.run.smithi032.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119711.log.gz 2024-08-31T22:56:13.207 INFO:teuthology.orchestra.run.smithi032.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103905.log.gz 2024-08-31T22:56:13.922 INFO:teuthology.orchestra.run.smithi032.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119930.log.gz 2024-08-31T22:56:14.001 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75403.log.gz 2024-08-31T22:56:16.414 INFO:teuthology.orchestra.run.smithi032.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.90607.log.gz 2024-08-31T22:56:16.446 INFO:teuthology.orchestra.run.smithi032.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.120412.log.gz 2024-08-31T22:56:16.750 INFO:teuthology.orchestra.run.smithi032.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.120276.log.gz 2024-08-31T22:56:17.105 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.91505.log.gz 2024-08-31T22:56:17.118 INFO:teuthology.orchestra.run.smithi032.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.120032.log.gz 2024-08-31T22:56:17.748 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50470.log.gz 2024-08-31T22:56:18.576 INFO:teuthology.orchestra.run.smithi032.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.90695.log.gz 2024-08-31T22:56:18.862 INFO:teuthology.orchestra.run.smithi032.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.119627.log.gz 2024-08-31T22:56:19.561 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41412.log.gz 2024-08-31T22:56:21.617 INFO:teuthology.orchestra.run.smithi032.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41302.log.gz 2024-08-31T22:56:22.092 INFO:teuthology.orchestra.run.smithi032.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.91411.log.gz 2024-08-31T22:56:22.579 INFO:teuthology.orchestra.run.smithi032.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.119497.log.gz 2024-08-31T22:56:22.638 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.120115.log.gz 2024-08-31T22:56:22.847 INFO:teuthology.orchestra.run.smithi032.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103814.log.gz 2024-08-31T22:56:24.502 INFO:teuthology.orchestra.run.smithi032.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62795.log.gz 2024-08-31T22:56:25.725 INFO:teuthology.orchestra.run.smithi032.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.120646.log.gz 2024-08-31T22:56:26.042 INFO:teuthology.orchestra.run.smithi032.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50363.log.gz 2024-08-31T22:56:31.279 INFO:teuthology.orchestra.run.smithi032.stderr: 94.6% -- replaced with /var/log/ceph/cluster2-client.mirror.29540.log.gz 2024-08-31T22:56:31.406 INFO:teuthology.orchestra.run.smithi032.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.29956.log.gz 2024-08-31T22:56:31.877 INFO:teuthology.orchestra.run.smithi032.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75302.log.gz 2024-08-31T22:56:42.231 INFO:teuthology.orchestra.run.smithi032.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-08-31T22:56:58.764 INFO:teuthology.orchestra.run.smithi032.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.30125.log.gz 2024-08-31T22:57:01.031 INFO:teuthology.orchestra.run.smithi032.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-client.mirror.1.29540.log.gz 2024-08-31T22:57:09.406 INFO:teuthology.orchestra.run.smithi032.stderr: 94.9% -- replaced with /var/log/ceph/cluster2-client.mirror.29512.log.gz 2024-08-31T22:58:13.279 INFO:teuthology.orchestra.run.smithi007.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-08-31T22:58:14.695 INFO:teuthology.orchestra.run.smithi032.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.29512.log.gz 2024-08-31T22:58:16.554 INFO:teuthology.orchestra.run.smithi007.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-08-31T22:58:35.231 INFO:teuthology.orchestra.run.smithi007.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-08-31T22:58:35.233 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-08-31T22:58:35.234 INFO:teuthology.orchestra.run.smithi007.stderr:real 2m40.240s 2024-08-31T22:58:35.234 INFO:teuthology.orchestra.run.smithi007.stderr:user 4m25.456s 2024-08-31T22:58:35.234 INFO:teuthology.orchestra.run.smithi007.stderr:sys 0m11.039s 2024-08-31T23:03:44.461 INFO:teuthology.orchestra.run.smithi032.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-08-31T23:03:51.435 INFO:teuthology.orchestra.run.smithi032.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-08-31T23:04:47.139 INFO:teuthology.orchestra.run.smithi032.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-08-31T23:04:47.140 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-31T23:04:47.140 INFO:teuthology.orchestra.run.smithi032.stderr:real 8m52.130s 2024-08-31T23:04:47.140 INFO:teuthology.orchestra.run.smithi032.stderr:user 10m20.792s 2024-08-31T23:04:47.140 INFO:teuthology.orchestra.run.smithi032.stderr:sys 0m31.061s 2024-08-31T23:04:47.142 INFO:tasks.ceph:Archiving logs... 2024-08-31T23:04:47.143 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-08-31_21:16:02-rbd-squid-distro-default-smithi/7883608/remote/smithi007/log 2024-08-31T23:04:47.144 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-31T23:05:09.374 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-08-31_21:16:02-rbd-squid-distro-default-smithi/7883608/remote/smithi032/log 2024-08-31T23:05:09.375 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-31T23:05:47.957 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-31T23:05:47.972 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-31T23:05:47.972 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-31T23:05:47.975 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-31T23:05:48.044 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-31T23:05:48.054 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-31T23:05:48.054 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-31T23:05:48.056 DEBUG:teuthology.orchestra.run.smithi032:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-31T23:05:48.092 INFO:teuthology.orchestra.run.smithi007.stdout: remote refid st t when poll reach delay offset jitter 2024-08-31T23:05:48.092 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================== 2024-08-31T23:05:48.093 INFO:teuthology.orchestra.run.smithi007.stdout:+hv01.front.sepi 67.205.162.81 3 u 42 128 377 0.110 -0.053 0.458 2024-08-31T23:05:48.093 INFO:teuthology.orchestra.run.smithi007.stdout:*hv02.front.sepi 63.231.80.2 3 u 110 128 377 0.117 -1.865 0.444 2024-08-31T23:05:48.093 INFO:teuthology.orchestra.run.smithi007.stdout:+hv03.front.sepi 74.6.168.72 3 u 40 128 377 0.147 -3.179 0.569 2024-08-31T23:05:48.093 INFO:teuthology.orchestra.run.smithi007.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-08-31T23:05:48.126 INFO:teuthology.orchestra.run.smithi032.stdout: remote refid st t when poll reach delay offset jitter 2024-08-31T23:05:48.126 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================== 2024-08-31T23:05:48.126 INFO:teuthology.orchestra.run.smithi032.stdout:+hv01.front.sepi 67.205.162.81 3 u 43 64 377 0.134 -0.325 0.337 2024-08-31T23:05:48.126 INFO:teuthology.orchestra.run.smithi032.stdout:*hv02.front.sepi 63.231.80.2 3 u 35 64 377 0.136 -2.563 0.308 2024-08-31T23:05:48.126 INFO:teuthology.orchestra.run.smithi032.stdout:+hv03.front.sepi 74.6.168.72 3 u 41 64 377 0.106 -3.699 0.260 2024-08-31T23:05:48.126 INFO:teuthology.orchestra.run.smithi032.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-08-31T23:05:48.127 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-31T23:05:48.135 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-31T23:05:48.136 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-31T23:05:48.144 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-31T23:05:48.152 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-31T23:05:48.160 INFO:teuthology.task.internal:Duration was 3791.727856 seconds 2024-08-31T23:05:48.160 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-31T23:05:48.169 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-31T23:05:48.169 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-31T23:05:48.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-31T23:05:48.370 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-31T23:05:48.370 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2024-08-31T23:05:48.371 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-31T23:05:48.381 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi032.front.sepia.ceph.com 2024-08-31T23:05:48.381 DEBUG:teuthology.orchestra.run.smithi032:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-31T23:05:48.392 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-31T23:05:48.392 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-31T23:05:48.423 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-31T23:05:48.499 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-31T23:05:48.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-31T23:05:48.503 DEBUG:teuthology.orchestra.run.smithi032:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-31T23:05:48.676 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-31T23:05:48.686 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-31T23:05:48.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-31T23:05:48.715 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-31T23:05:48.754 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-31T23:05:48.763 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-31T23:05:48.766 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-31T23:05:48.787 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2024-08-31T23:05:48.819 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = core 2024-08-31T23:05:48.849 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-31T23:05:48.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T23:05:48.858 DEBUG:teuthology.orchestra.run.smithi032:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-31T23:05:48.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T23:05:48.896 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-31T23:05:48.906 INFO:teuthology.task.internal:Transferring archived files... 2024-08-31T23:05:48.906 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-31_21:16:02-rbd-squid-distro-default-smithi/7883608/remote/smithi007 2024-08-31T23:05:48.907 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-31T23:05:48.928 DEBUG:teuthology.misc:Transferring archived files from smithi032:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-31_21:16:02-rbd-squid-distro-default-smithi/7883608/remote/smithi032 2024-08-31T23:05:48.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-31T23:05:48.960 INFO:teuthology.task.internal:Removing archive directory... 2024-08-31T23:05:48.960 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-31T23:05:48.967 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-31T23:05:49.008 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-31T23:05:49.026 INFO:teuthology.task.internal:Not uploading archives. 2024-08-31T23:05:49.026 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-31T23:05:49.035 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-31T23:05:49.036 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-31T23:05:49.038 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-31T23:05:49.042 INFO:teuthology.orchestra.run.smithi007.stdout: 658468 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 31 23:05 /home/ubuntu/cephtest 2024-08-31T23:05:49.056 INFO:teuthology.orchestra.run.smithi032.stdout: 658467 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 31 23:05 /home/ubuntu/cephtest 2024-08-31T23:05:49.057 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-31T23:05:49.068 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-31T23:05:49.090 INFO:teuthology.run:Summary data: description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} duration: 3791.727855682373 failure_reason: '"2024-08-31T22:15:38.560806+0000 mon.a (mon.0) 240 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY)" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-08-31T23:05:49.091 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-31T23:05:49.175 INFO:teuthology.run:FAIL